Workflow: Nothing Moves by Itself

What Moves by Itself, and What We Move

Pinning alone is half of it: the story’s chart was pinned - and two releases old. The other half is a schedule.

The Dashboard: What Is Out of Date

Renovate reads the repository and knows where versions live: Helm charts in HelmReleases, Terraform providers and modules, GitHub Actions. Versions outside such files carry a comment that says where to look them up:

# renovate: datasource=github-releases depName=k3s-io/k3s versioning=...
k3s_version = "v1.36.4+k3s1"

On this platform it runs as a workflow on the 1st of every month and opens no pull requests (dependencyDashboardApproval). It keeps one issue - Platform maintenance: dependency updates - listing every pinned version with a newer release. The first run listed about twenty: Kubernetes one minor behind, the monitoring stack ten major versions behind, the database operator two releases behind, Flux one patch behind.

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "description": [
    "Dependency updates on our own schedule (Chapter 20): Renovate opens no pull requests. It keeps one",
    "issue - the dependency dashboard - listing what is out of date; updates are made by hand once a month,",
    "after reading the release notes: patch versions together in one pull request, every minor or major",
    "version in one per component (the chart with the tool that tests it).",
    "Versions outside a standard file carry a '# renovate: datasource=... depName=...' comment above them.",
    "Vulnerability alerts stay with GitHub's own alerts (on for the repository); the workflow token cannot read them."
  ],
  "extends": [
    "config:recommended"
  ],
  "dependencyDashboard": true,
  "dependencyDashboardTitle": "Platform maintenance: dependency updates",
  "dependencyDashboardApproval": true,
  "vulnerabilityAlerts": {
    "enabled": false
  },
  "flux": {
    "managerFilePatterns": [
      "/^flux/.+\\.ya?ml$/"
    ]
  },
  "customManagers": [
    {
      "customType": "regex",
      "description": "Versions pinned in Terraform variables, tfvars, the Makefile, CI workflows and Flux manifests (where no standard manager reads them), each under a '# renovate:' comment.",
      "managerFilePatterns": [
        "/(^|/)Makefile$/",
        "/\\.tf$/",
        "/\\.tfvars$/",
        "/^\\.github/workflows/.+\\.ya?ml$/",
        "/^flux/.+\\.ya?ml$/"
      ],
      "matchStrings": [
        "#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+registryUrl=(?<registryUrl>\\S+))?(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n\\s*[A-Za-z_][A-Za-z0-9_]*\\s*(?::=|:|=)\\s*\"?(?:[^\"\\s:@]+:)?(?<currentValue>[^\"\\s@#]+)(?:@(?<currentDigest>sha256:[a-f0-9]+))?\"?"
      ]
    }
  ],
  "packageRules": [
    {
      "description": "The platform's own images are updated by Flux image automation, not here.",
      "matchPackageNames": [
        "/^ghcr\\.io/safeops-course//"
      ],
      "enabled": false
    },
    {
      "description": [
        "origin-ca-issuer's CRDs come from its Git repository pinned by commit (flux/bootstrap/components/public-edge/public-edge.yaml).",
        "Renovate would follow the default branch's HEAD, not the release tags; the chart version (helm datasource) is the signal",
        "for a new release, and the CRD commit is moved to that release's tag by hand, in the same pull request."
      ],
      "matchDatasources": [
        "git-refs"
      ],
      "matchPackageNames": [
        "https://github.com/cloudflare/origin-ca-issuer"
      ],
      "enabled": false
    }
  ]
}

Why not a pull request per update? Each would be merged on its own, between other work, often without its release notes read. This platform updates once a month, on one day: the dashboard says what is due, a person reads what changes, the versions that belong together move together. How they are split into pull requests depends on what changes:

• patch versions - fixes, no new behaviour - go together in one pull request;
• a minor or major version, or anything whose release notes announce a change in behaviour, gets a pull request of its own, one per component - the chart together with the tool that tests it (the Kyverno chart and the Kyverno CLI).

A component’s own pull request that fails points at that component, and its revert takes back one change, not the whole month. The patch pull request still holds several components - accepted, since patches are fixes, and the risky steps are not in it. The story’s upgrade - two minor versions, three problems at once - is the case for the second rule. The issue is also the reminder for node maintenance - the same day.

The platform’s own images are excluded: Flux image automation updates them, and they are this platform’s releases, not dependencies.

One Update, Step by Step

1. Read the release notes of every version between the pinned one and the target - not only the target’s. Removals and changed defaults are announced in the version that makes them.
2. Render what changes. helm template of the old and the new chart, compared: new images, new objects, removed ones. It shows what the chart does; the release notes show what the software does.
3. Move what belongs together together. A chart and the tool that tests it (the Kyverno CLI for the Kyverno chart, promtool for Prometheus) in the same pull request.
4. Let the checks run - the policy tests, the alert tests, and the two checks this chapter adds (below).
5. Merge, and watch the upgrade finish. Flux runs helm upgrade, Helm waits until every pod is Ready (timeout, 15 minutes for Kyverno: image pulls on a slow network took most of the 5-minute default), and a failed upgrade is rolled back (remediation). A rollback puts the release back, not necessarily the platform: CRDs applied by the chart, what its hooks did and changes in data can stay. After one, check the platform works - then explain the failure.
6. Check what it does, not only that it is Ready: the lab of the chapter the component belongs to, on kind first.

Small steps: one minor at a time for Kubernetes (its components support only a limited version skew), and the monthly rhythm for everything else - a month of changes is readable; two years is not.

Tests That Run What Runs

• tests/tool-versions.test.sh asks Helm what the pinned charts install - the Kyverno chart’s appVersion, the Prometheus image tag of kube-prometheus-stack - and compares them with the Kyverno CLI and promtool pinned in CI. A chart bumped without its tool, or the other way round, fails the pull request: the tests run 1.19.1, the chart in Git installs 1.17.1. It compares with Git, not with the cluster: after an upgrade that failed and was rolled back it stays green - what runs is read from the cluster.
• kubeconform checks against our Kubernetes version - read from kind’s node image - not against the newest schemas. A probe stops the check when the schemas for that version (or Flux’s) are missing: with -ignore-missing-schemas, a missing schema set would otherwise turn every check into a silent skip.

Before a Kubernetes Upgrade: Removed APIs

Every Kubernetes minor removes APIs that were deprecated a few releases earlier. An object still written for a removed API is refused by the upgraded cluster - Flux cannot apply it any more. Two places to look, before the upgrade:

What Git holds. scripts/flux-kustomize-validate.sh validates every manifest against a Kubernetes version - the one you upgrade to, when you name it:

FLUX_VALIDATE_KUBERNETES_VERSION=1.37.0 scripts/flux-kustomize-validate.sh

One trap, found while writing this chapter: the check runs with -ignore-missing-schemas, because the CRDs we use (Kyverno, CloudNativePG, the monitoring stack) have no published schemas. That flag skipped an object of a removed API too - no schema, Skipped: 1, exit 0. Kubernetes’ own API groups always have schemas, so for them a missing schema means a removed API: they are now checked a second time without the flag, and fail - a Kubernetes API that 1.36.4 does not serve.

What the cluster is asked for. Objects that are not in Git - what Helm charts render, what controllers request - and the versions of CRDs are invisible to that check. The API server marks every deprecated API that has been requested since it started, in the gauge apiserver_requested_deprecated_apis (value 1 per API, not a count of requests), with the release that removes it (removed_release) - and Prometheus already collects it. On this platform it lists no API scheduled for removal. It does list Kyverno’s kyverno.io/v1 ClusterPolicy as deprecated - the API Chapter 16’s admission guardrails used until they moved to ValidatingPolicies; it stays in the list because Kyverno itself still watches it. The gauge names the API, not the client: it is the start of a question - which objects, which caller - not the answer.

Stopping a Pod With Fewer Dropped Requests

Every rollout and every drain stops pods, and stopping has a race: the pod gets SIGTERM while the Service’s endpoints and Traefik still send it requests - they learn that it is going a moment later. The backend closes its listener at once; nginx exits on SIGTERM. Whatever arrives in between fails.

Measured on kind - a rollout of two replicas under about 115 requests a second:

The fix is a wait before the stop: lifecycle.preStop.sleep.seconds: 5. The kubelet waits five seconds before it sends SIGTERM; the endpoints and Traefik catch up while the pod still serves; then the app shuts down cleanly. It is the kubelet’s own sleep - the image needs no shell. Ten seconds did no better than five here. The wait counts against the grace period (30 seconds): five seconds plus the backend’s ten-second shutdown fit. scripts/check-app-security.sh fails an app container without it.

A drain stops pods the same way, so node maintenance gets the same protection - where there is a second node to move them to.

Node Maintenance

On the Hetzner cluster, scripts/node-maintenance.sh updates one node, at a time chosen and announced:

1. Dry run first (the default): a server-side drain dry run - PodDisruptionBudgets are checked - that names what would block.
2. --apply: out of the load balancer (below), cordon, drain, transactional-update over ssh (the OS installs into a new snapshot), reboot, wait for a new boot ID and Ready, uncordon, back into the load balancer.
3. If a PodDisruptionBudget blocks the drain, the script stops. --apply --accept-outage is the decision to take the outage: pods are deleted past their budgets and come back with the node.

Measured on the Hetzner cluster (2026-10-11): --apply on a worker took 217 seconds from cordon to uncordon - the drain, the OS update into a new snapshot, the reboot, the wait for a new boot ID. A probe from outside (one request a second to the staging site) saw about 22 seconds of timeouts during that reboot, although no application pod was left on the node. The cause is the load balancer, which keeps sending traffic to a node until its health check fails: a TCP check every 15 seconds, a 10-second timeout, three failures - up to about a minute before a rebooting node is taken out (the next cycle read the settings). A cordon does not help: since Kubernetes 1.21 it no longer removes a node from a load balancer. So the script now takes the node out first - the label node.kubernetes.io/exclude-from-external-load-balancers, which tells the cloud controller to drop it from every Service of type LoadBalancer - waits a minute, and puts it back after the uncordon. The script cannot see the load balancer, so that minute is a wait, not a proof; the proof is the same probe from outside on the next Hetzner cycle, with the timeouts gone.

With one worker, the drain cannot complete: the pods have nowhere to go, and CloudNativePG’s budget keeps the primary (app-postgres-primary, 0 disruptions allowed, in every environment) - in develop and staging their only instance; production’s replica may be evicted, its primary not. A second worker does not change that for the databases - the budget, not the capacity, blocks them; it was tried. CloudNativePG has its own switch for it: spec.nodeMaintenanceWindow: {inProgress: true, reusePVC: true} on the Cluster drops the budget. The switch moves nothing by itself: the drain then evicts the instance, and it starts again on another node with the same volume - if that node can attach the volume and has room for the pod. On Hetzner that took about four and a half minutes per database, most of it waiting for the cloud volume to detach from the old node. --apply --accept-outage on the only worker, with all three databases on it (2026-10-12): 291 seconds from cordon to uncordon, and the databases came back by themselves about a minute and a half after the reboot, each on its own volume. The users’ side, measured from outside: develop down about 5 minutes - and production 18. Both Kyverno admission controllers had run on that worker; while it rebooted, the fail-closed signature check refused every new pod, and the ReplicaSets, backing off between retries, came back to production long after Kyverno had. One fix, in Chapter 16’s terms: the admission controller spread one per node - it tolerates the control plane’s taint, which stays on for everything else (allow_scheduling_on_control_plane = false) - to be measured again on the next cycle. So maintenance here is a planned outage - in a window, announced, short. That is the honest cost of one worker - with both of production’s database instances on it, a second instance does not help when the node goes - acceptable for a course cluster that exists for a few hours. It is not a production design: production needs at least two workers, so drained pods have somewhere to run - Chapter 09 told the same story from the drain’s side, “two replicas are not two nodes” - and a database that survives losing one of them (Disaster Recovery, Chapter 22).

Kubernetes Itself

k3s_version pins what a new Hetzner cluster installs. kube-hetzner also renders it into the plans of the system-upgrade-controller, which upgrades nodes in place - but those plans only select nodes labelled k3s_upgrade=true, a label the module sets only when auto_upgrade_k3s is on. With it off, an in-place upgrade means: the new version in a pull request and an apply, then the label on one node at a time, by hand.

That path was run on the Hetzner cluster (2026-10-11, v1.37.0 → v1.37.1):

• the apply rendered the new version into both plans, and nothing moved until a node was labelled;
• the control plane upgraded in 62 seconds, with no failed request seen from outside;
• the worker never finished: the plan drains with eviction, the database budgets refused it, and the job retried “Cannot evict pod as it would violate the pod’s disruption budget” for as long as anyone let it - the node cordoned, nothing alerting. The platform alerts on it now: NodeCordonedTooLong, a node cordoned for more than 30 minutes - a maintenance takes about five. Moving the databases with nodeMaintenanceWindow (above) let it complete.

kube-hetzner’s own advice for small clusters is to turn eviction off for these upgrades, “to avoid system upgrades hanging since pods resisting eviction keep nodes unschedulable forever”. This platform now does (k3s_upgrade_eviction = false): an upgrade deletes the pods it drains, past every PodDisruptionBudget on that node - the applications’ as well as the databases’. For develop’s and staging’s single-instance databases that is the same short planned outage as node maintenance. Turn it back on when every database survives losing an instance. The setting itself is checked on the next Hetzner cycle; until then, label a node that runs a database only with the databases moved first.

kind’s node image is pinned by digest (infra/terraform/kind_cluster); a new one recreates the local cluster - its UPGRADE.md lists the steps. Every cluster runs the same Kubernetes minor, 1.36. kind and Hetzner pin the exact same version, 1.36.4; a managed cloud runs whichever 1.36 patch its provider offers. The policies and charts are tested on kind’s 1.36.4 - a provider’s patch is not, so on a managed cloud make cluster-conformance checks the minor (C1) and the smoke test checks the rest. The removed-API check reads kind’s pin by default; removed APIs change between minors, not patches. 1.37 waits for kind: it needs the kubeadm v1beta4 configuration, which kind writes from v0.32 on, and the Terraform provider that creates the kind cluster still embeds kind v0.31. When the provider follows, every cluster moves together - run the check with FLUX_VALIDATE_KUBERNETES_VERSION=1.37.1 first, as above.

What This Does Not Solve

• A security fix that cannot wait a month. The monthly rhythm is for routine updates; a critical CVE in a running component (Chapter 17’s SBOMs say which images contain it) is an update the same day, through the same steps.
• Upgrades without an outage on this cluster. One worker - with every database instance on it - means planned downtime for node maintenance - see above.
• The release notes themselves. Nothing automates reading them; the checks catch what they can - versions that differ, schemas that do not match - not a changed default.

Instructions for Your AI Agent

The rules this chapter adds:

1. Never unpin a version (a range, latest, a major tag) and never turn on automatic updates for charts, nodes or k3s: every version moves through a reviewed pull request. Enforced by: review; scripts/check-image-tags.sh for the app images; Renovate’s dashboard shows what is due.
2. Bump a chart and the tool that tests it in the same pull request; never edit a test’s tool version to make a mismatch pass. Enforced by: tests/tool-versions.test.sh (pre-commit and CI).
3. Read the release notes of every version between the pinned one and the target, and render the chart diff, before proposing an upgrade - and say what changes in the pull request. Enforced by: instruction; review.
4. Never drain or reboot a node without the dry run first and the owner’s yes for the window; never use --accept-outage on your own. Enforced by: scripts/node-maintenance.sh defaults to a dry run; instruction.
5. Every app container keeps its preStop sleep; never remove it to make a rollout faster. Enforced by: scripts/check-app-security.sh (pre-commit and CI).
6. Before proposing a Kubernetes minor, run the removed-API checks against the target version and report what they found. Enforced by: instruction; scripts/flux-kustomize-validate.sh fails a removed API.

This builds on: Availability & Scaling (Chapter 09) - PodDisruptionBudgets and drains - and Supply Chain Security (Chapter 17): the policy engine whose upgrade is this chapter’s story. This enables: Credential & Access Lifecycle (Chapter 21) - the same discipline for credentials: everything rotated on a schedule, nothing left to age.