Investigation & Containment

Investigation

Five questions, in this order: what leaked, what it opens, where every copy lives, what reads it, and whether the old value is really dead. Every kubectl names the cluster with --context (Chapter 01).

  1. What leaked? Not “a file” - the credentials in it, one by one. In the story the artifact was a saved plan, and in 2026-09 a saved plan held every Terraform input variable in plain text: the Hetzner token, the node SSH key and the age key. Whether a value is in a plan depends on how it is declared, so read the declarations, not only the list of TF_VAR_s:

    grep -n 'TF_VAR_' .github/workflows/terraform-hcloud.yml
grep -n -B4 'ephemeral *= *true' infra/terraform/hcloud_cluster/variables.tf

    sensitive = true only hides a value in the plan’s output - it is still in the saved plan and the state. ephemeral = true keeps it out of both. Today only the age key is ephemeral; the SSH key no longer passes through Terraform at all (it signs in through ssh-agent). The Hetzner token, the backup keys and the other tokens are still in every saved plan - which is why the plan is now encrypted before it is uploaded.

  2. What does each one open? The blast radius is not the credential, it is what it can reach. The registry says it in one line per credential:

    yq '.credentials[] | .name + ": " + .what' docs/credential-registry.yaml

    The age key’s line is the widest: decrypts every Secret under flux/secrets/ except local/ - every copy encrypted for it, in every commit until the key is replaced. Files encrypted only for a new key are out of its reach; everything before is not. A leaked age key is a leak of every value it protected - so all of them are rotated, not only the key.

  3. Where does every copy live? A rotation that misses a copy either keeps the old value alive there, or breaks the first time that copy is used. lives_in lists them - CI secrets, SOPS files, Secrets that Terraform writes into the cluster, a person’s workstation:

    yq '.credentials[] | select(.name == "hcloud-token") | .lives_in' docs/credential-registry.yaml

    For the Hetzner token: the organization secret, the owner’s hcloud context - and two Secrets in the cluster’s kube-system, which kube-hetzner writes for the cloud controller and the CSI driver. The last two are easy to forget: nobody put them there by hand.

  4. What reads it, and when? A new value in a Secret reaches nothing by itself. A pod that reads a Secret as environment variables reads it once, at start; until it restarts, it uses the old value - and after the old one is revoked, it fails. used_by names the readers; the Deployment says how they read:

    kubectl --context kind-sre-control-plane -n develop get deploy backend \
  -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name} <- {.valueFrom.secretKeyRef.name}{"\n"}{end}' | grep -v '<- $'

    The backend reads JWT_SECRET, the database password and the telemetry DSN from Secrets - all three only at start.

  5. Is the old value really dead? The question the story skipped. A rotation ends with a test that uses the old value and is refused - an API call with the old token, a login with the old password. In the story nobody asked, and the leaked Hetzner token stayed valid for sixteen days after a “rotation”.

The root cause is not the artifact. It is credentials that nobody listed, so their rotation was worked out under pressure - and ended when the new value worked, not when the old one was refused.

Containment

  1. Revoke first, if the holder can act now. For a token that can delete the platform, a short outage while CI gets the new value is cheaper than another hour of exposure.
  2. Replace everything the leaked credential could read. After an age key leak: the key, and every value behind it - at its source (a new OAuth secret, a new token), not only re-encrypted.
  3. Update every copy, restart every reader, then revoke. In that order, so nothing breaks in between - except where step 1 decided otherwise.
  4. Prove both halves: the new value works, the old one is refused.
  5. Write it down - the registry gets the date, and anything left on purpose gets a reason and an end date (accepted), not silence.

A rotation is done when the old value is refused - not when the new one works.

Pause and Predict: A database password is replaced in its Secret, and the database accepts the new one. The backend pods were not restarted. What do they do with requests that need the database - now, and after the old password is refused?