Workflow: Every Credential Has a Date

The Registry

docs/credential-registry.yaml lists every credential the platform runs on - names and dates only, never a value. One entry:

- name: postgres-app-develop-staging
  what: the app role of the develop and staging databases
  class: key
  lives_in:
    - flux/secrets/develop/cnpg-postgres-secret.yaml
    - flux/secrets/staging/cnpg-postgres-secret.yaml
  used_by: CloudNativePG (creates the role), backend and its migrations
  rotated: 2026-09-26
  runbook: docs/credential-rotation.md#database-passwords

scripts/check-credentials.sh reads it and fails:

• two weeks before a rotation is due, or before expires - while there is still time to do it calmly;
• for an exposed value not replaced since, unless accepted holds it back - and for accepted past its date, longer than 30 days, or accepting nothing;
• for a runbook heading that does not exist;
• for every key of every SOPS file under flux/secrets/ that no entry names - a new Secret, or a new key in an existing one, cannot skip the registry.

It runs in pre-commit and CI, and every morning on main (.github/workflows/credential-rotation.yml): a date passes without a commit, so a check that runs only on pull requests would never notice it. The dates are SafeOps’ own: in a fork - your copy - its credentials are not yours to rotate, so CI there only reports what is due (CHECK_CREDENTIALS_DATES=report) and the morning run does not start. Everything else - a missing description, runbook or registry entry - fails everywhere. Two limits of that reminder:

• A failed scheduled run e-mails one person - whoever last changed its schedule - not the team.
• A run that never happens reports nothing: GitHub disables scheduled workflows in a public repository after 60 days without activity, and a disabled workflow is silent.

So the reminder has a human backstop: the monthly maintenance day of Chapter 20 starts by checking that the scheduled workflows ran (Actions → Credential rotation, the last runs green and recent). Where a provider sets its own end date, expires records it, and a credential that outlives its reminder still ends - but only those credentials have that backstop; most here have no provider expiry.

The local kind cluster’s credentials are not in the registry - the age key and passwords Terraform generates for it are throw-away. The registry is the platform’s: the Hetzner cluster, CI, Cloudflare, GitHub.

A Rotation, Step by Step

docs/credential-rotation.md has one section per credential; every one follows the same six steps:

1. Create the new value next to the old one where the provider allows two at a time - a GitHub OAuth App keeps two client secrets, Hetzner and R2 tokens and deploy keys can coexist. Where it does not - a Cloudflare token “roll”, a password - plan the short gap.
2. Update every place in lives_in. Values never pass through a command line, a chat or an issue: gh secret set reads them from a prompt, sops edit from an editor.
3. Restart the readers - through Git (next section).
4. Prove the new value works - the check the runbook names: a login, an API call, a backup that completes.
5. Revoke the old value, and prove it is refused. The step the story skipped. What “refused” can mean depends on the kind of credential (below).
6. Set rotated to today, in the same pull request.

The second proof depends on what the credential does. Not every old value can be refused:

A decryption key is the case the story got wrong at first: replacing the key protected new files, not the history.

After a leak, the order changes in one place: if the holder can do damage now, revoke first and take the short outage.

Restarting the Readers Through Git

A Secret that changes restarts nothing. The usual answer is kubectl rollout restart deployment/backend - and on a Deployment that Flux manages, it can silently do nothing:

• rollout restart sets an annotation (kubectl.kubernetes.io/restartedAt) on the pod template. That annotation is not in Git.
• Flux applies what Git says and removes fields changed by hand on objects it manages - at its next reconcile, the annotation is gone.
• If that reconcile comes while the rollout runs, the template is back to the old one: the new pods are scaled away, the old ones stay - with the old value. rollout status even reports success: the Deployment is, again, what Git says.

So the restart goes through Git, like every other change of a Flux-managed object. Every reader has an annotation on its pod template, and a rotation sets it to the current time in the same pull request:

# flux/apps/backend/develop/kustomization.yaml - a patch on the backend's pod template
annotations:
  safeops.io/secrets-rotated: "2026-10-11T13:30:00Z"   # date -u +%FT%TZ

The template changes, and Flux rolls the pods like any release - a new pod Ready before an old one goes. A timestamp, not a day: two rotations on one day must still change the template. It is the same idea as Chapter 18’s feature flags, whose generated ConfigMap name changes with its content.

What Flux does not manage - the Hetzner cloud controller in kube-system, deployed by kube-hetzner - keeps kubectl rollout restart.

The Database Password

The app’s database role is a CloudNativePG managed role: its password comes from the Secret app-postgres-app (spec.managed.roles[].passwordSecret). When the Secret changes, CloudNativePG sets the new password in PostgreSQL itself and records which version of the Secret it applied:

kubectl --context kind-sre-control-plane -n develop get secret app-postgres-app -o jsonpath='{.metadata.resourceVersion}{"\n"}'
kubectl --context kind-sre-control-plane -n develop get cluster app-postgres \
  -o jsonpath='{.status.managedRolesStatus.passwordStatus.app.resourceVersion}{"\n"}'

Equal numbers: the database has the new password - on kind, about four seconds after the change. From that moment the old password is refused for every new connection; connections already open stay open. The backend reads the password at start, so its running pods keep the old one until the rollout - and every new pod’s migrate init container signs in to the database before the backend starts: a wrong password stops the rollout, and the old pods keep serving.

The order. The Secret and the restart arrive in one commit, but two Kustomizations apply it, each on its own schedule: secrets-<env> the Secret, backend-<env> the restart. Had the backend rolled first, its new pods would have started with the old value and kept it until the next rollout - a rotation that looks done and is not. So backend-<env> depends on secrets-<env>: for a dependency on the same Git source, Flux waits until the dependency has applied the same revision. The restart of a commit always comes after its Secret. CloudNativePG then needs a few seconds more to change the password; a new pod that starts in between fails at migrate and is retried, while the old pods keep serving.

The gap. Between the moment CloudNativePG applies the new password and the moment the last old pod is replaced, the old pods can use only the connections they already hold. A request that needs a new connection - after an idle one closed, or under more load - fails. With the Secret and the restart in one pull request, the gap is the Flux reconcile that picks up the commit plus the rollout: seconds to a few minutes here. This platform accepts it and keeps it short. Closing it needs two database roles used in turn - the app moves to the second role’s credentials, then the first role’s password changes while nothing uses it - which CloudNativePG supports and this platform does not set up. Where the gap is not acceptable, that is the design to use.

People: Access by Team

On the Hetzner cluster, people sign in through Dex, an OIDC provider in front of GitHub. Only members of two GitHub teams can sign in; their team becomes a group in the token, and RBAC binds roles to the group:

The last column is a ValidatingAdmissionPolicy (oidc-flux-operator-fields): admins may pause or kick Flux during an incident, but every other change to a Kustomization or HelmRelease is refused - it goes through Git. Writing anywhere else is the break-glass kubeconfig.

RBAC alone would not hold the members’ “read” either. admin in develop includes Flux’s own edit roles - Flux aggregates them into admin - and Flux applies what a Kustomization names with cluster-admin rights: one Kustomization created in develop would be cluster-admin for whoever created it. A second policy, oidc-flux-objects-from-git, refuses every create, change or delete of a Flux object by these groups, in every namespace; the admins’ suspend and reconcile of a Kustomization or HelmRelease in flux-system are its only exception. A role is more than the rules it lists: aggregation can add to it later.

kind runs no Dex, but it has the same roles - and impersonation asks the API server what a person in a group may do, without signing in as one:

kubectl --context kind-sre-control-plane auth can-i delete pods -n production \
  --as=[email protected] --as-group=safeops-course:members      # no

Taking access away is removing the person from the team - and it is not instant. The API server trusts a token until it expires; there is no revocation list for OIDC tokens. Dex’s default lifetime is 24 hours: someone removed from the team kept access for up to a day. This platform sets it to one hour: after that the API server refuses the token, and the client has to get a new one. Nothing refreshes it by itself - the client (kubelogin, Headlamp) redeems its refresh token or signs in again when it next needs one, and that is when Dex reads the person’s teams from GitHub again: out of the team, no group, no role. So a team change reaches a token only when the client asks for a new one; what bounds the old access is the hour. (A token issued before a change of the lifetime keeps the lifetime it was signed with.) The hour comes from Dex’s configuration and its source code; removing a test account and timing it on the live cluster is still to be done.

Break-Glass Access

The kubeconfig Terraform writes holds a cluster-admin client certificate. It is for when OIDC is not enough - Dex is down, or an incident needs a write no role allows:

• who: the person who applied Terraform, on that machine, mode 0600, never copied;
• when: said where the team talks before it is used - who, why, for how long;
• after: what was changed is written down, and anything that should stay goes back into Git (or Flux undoes it).

A client certificate cannot be revoked on its own - Kubernetes has no revocation list for them. It ends when it expires or when the cluster’s certificate authority is replaced. On the Hetzner cluster (k3s, checked 2026-10-11) the admin certificate - CN=system:admin, group system:masters - is valid for one year, and the authority that signs it for ten: a copy that leaks stays good for up to a year. This platform’s Hetzner cluster is rebuilt for every test cycle, which replaces both. On a cluster that lives on, a lost admin kubeconfig means rotating the CA - an outage to plan. RBAC cannot shorten that wait: the certificate’s group, system:masters, is allowed everything by the API server before any RoleBinding is read, so deleting a binding changes nothing. An admission policy that refuses requests from system:admin can stop its writes - not its reads, and not without locking out every other copy of the same certificate, yours included. That stopgap has not been rehearsed here. That is why people get roles that can be removed, and the certificate stays break-glass.

A Secret by Content

Chapter 04’s checks look at file names (no-secrets) and SOPS structure (sops-encrypted). A token pasted into a values file or a script passes both. gitleaks reads the content: a pre-commit hook on the staged changes, and a CI job that scans the whole history - a secret removed in a later commit is still in Git. Its rules are the defaults, with no path allowlist: a path allowlist hides real tokens under that path. A planted test value is allowed one line at a time, with an inline gitleaks:allow comment.

What This Does Not Solve

• Knowing whether a leaked credential was used. The providers’ audit logs (Hetzner, Cloudflare, GitHub) are the place to look; this platform does not collect them.
• Rotation without a person. Every rotation here is done by hand, on a date the check enforces. Dynamic secrets (Vault, OpenBao) or workload identity remove whole classes of long-lived credentials - see the alternatives in the chapter overview.
• A single platform age key for all three environments - Chapter 04 explains why one key is honest when one cluster decrypts all of them.
• Least privilege at the providers. A Hetzner token is read-write on a whole project; the scope of each token is the provider’s, and the registry records it, not limits it.

Instructions for Your AI Agent

The rules this chapter adds:

1. Never put a credential’s value into a command line, a file outside flux/secrets/ (SOPS), a chat, an issue or a pull request. Enforced by: gitleaks (pre-commit and CI), no-secrets, sops-encrypted.
2. Every new credential, and every new key in a SOPS Secret, gets a registry entry with a runbook in the same pull request. Enforced by: scripts/check-credentials.sh (pre-commit, CI).
3. Restart readers of a rotated Secret through Git (safeops.io/secrets-rotated), never with kubectl rollout restart on a Flux-managed Deployment. Enforced by: instruction; Flux removes the annotation rollout restart adds.
4. A rotation is reported done only with both proofs: the new value works, the old one is refused. Enforced by: instruction; the runbooks.
5. Never use or copy the break-glass kubeconfig without the owner’s yes for that incident. Enforced by: the file exists only on the machine that applied Terraform; instruction.

This builds on: Secrets Management (Chapter 04) - SOPS, the age key and its rotation - and Platform Lifecycle (Chapter 20): the same discipline, nothing left to age. This enables: Disaster Recovery (Chapter 22) - rebuilding the platform when the cluster is lost needs every credential, and the registry says where each one is.