The Incident: Three Keys in a Public Artifact

Incident Hook

This one happened on this platform.

The Terraform workflow of the platform repository saved its plan and uploaded it as a CI artifact, so the apply job could apply exactly what had been reviewed (Chapter 02). The repository is public; so were its artifacts. And a saved plan holds every input variable in plain text - Chapter 02 says it about the state, and it is just as true of the plan.

On 2026-09-25 a code review found it. The live artifact held:

  • the Hetzner API token - read and write on the whole project: create and delete servers, networks, volumes;
  • the node SSH private key - root on every node;
  • the SOPS age key - the key that decrypts every Secret under flux/secrets/ except local/ (each learner’s own key): every copy encrypted for it, in every commit, until the key is replaced.

Anyone who downloaded that artifact could do all three. Whether anyone did, nobody could say.

What followed, in order:

  1. The artifact was deleted. It did not undo anything: whoever had a copy kept it.
  2. The Hetzner token was rotated the same evening - a new one in the CI secret.
  3. The SSH key was rotated - a new key pair for the nodes of the next cluster.
  4. The plan artifact was encrypted with a passphrase, and the production database password is now generated by Terraform instead of kept in Git.
  5. The age key was replaced - and so was every value it protected. Re-encrypting with a new key was not enough: the old key still opens every encrypted file already in the Git history. So every SOPS value was replaced at its source: a new GitHub OAuth App for Dex, a new Cloudflare DNS token, three new JWT signing secrets, new database passwords, a new deploy key for image automation. One value - the observability ingest DSN - was left as it was, on purpose, as debt.

It took two days. Most of it was not typing; it was finding things out: which credentials there are, where each one is copied, what reads it, and how to replace it. A runbook existed for one of them - the age key.

Two weeks later, while this chapter was being written, one more thing came out: the leaked Hetzner token had never been deleted. A new one had been created and put into CI, and the old one was still valid in the Hetzner console - for sixteen days after the leak, any copy of that artifact could still delete every server. It was deleted on 2026-10-11.

Result: three keys public for an unknown time; two days of rotation worked out under pressure; one “rotated” token still alive two weeks later.

Observed Symptoms

  • A public CI artifact holding credentials in plain text - a saved plan, which nobody thought of as a secret.
  • No list of the platform’s credentials: each one had to be found by searching workflows, Terraform variables, SOPS files and the provider consoles.
  • A rotation that ended when the new value worked - nobody checked that the old one was refused.
  • After the rotation, no record of when each credential had last been replaced, or when it was due again.

Confusion Phase

The first explanations made it smaller than it was:

  • “The artifact is deleted, so the leak is closed.”
  • “The age key only matters for new files - we have a new one now.”
  • “The new token works, so the rotation is done.”

Each was half true. The artifact was gone - and so was any way to know who had copied it. The new key protected new files - and the old one still opened the whole history. The new token worked - and the old one did too.

What AI Would Propose (Brave Junior)

  • “Delete the artifact and rotate the key that matters most; the rest is unlikely to have been seen.”
  • “Re-encrypt everything with the new age key - sops updatekeys on every file.”
  • “Rotate everything every 24 hours with a CronJob, then a leak never matters.”
  • “Give everyone the admin kubeconfig, then nobody waits during an incident.”

Why This Is Dangerous

  • A deleted copy is not a revoked credential. Whoever has the value keeps it until the provider refuses it. Rotation ends with the old value revoked, and a test that proves it.
  • A leaked key opens everything it ever protected. sops updatekeys re-encrypts the data key for the new recipient; the values, and the old copies in Git history, stay readable with the old key. After a leak, every value behind the key is replaced at its source.
  • Rotation without a list is guesswork. Without knowing where every copy lives, a rotation leaves one behind - which then either keeps the old value alive, or breaks the first time it is used.
  • Automatic rotation needs readers that follow. A value replaced every night is only safe if every reader picks it up every night - a pod that read it at start keeps the old one.
  • Shared admin access cannot be taken back. A kubeconfig with a client certificate works until it expires; Kubernetes cannot revoke one certificate. Give people roles that can be removed, and keep the admin certificate for break-glass.

Pause and Predict: A token leaked. You created a new one, put it everywhere it is used, and the app works with it. What one check is still missing before the rotation is done - and what happens if you skip it?