Workflow: Seconds for a Pod, Minutes for a Cluster

What Can Be Lost, and What Brings It Back

Recovery is planned per failure. For each failure, two numbers: how long until it works again, and how much data can be lost. They are not the objectives (RTO, RPO) - they are what this platform delivers, to compare with them. The recovery times were measured on kind; the Hetzner cluster adds its own creation time to the last rows. The data loss was not measured - it follows from asynchronous replication and WAL archiving, and some rows have no upper bound.

Two things this table makes plain. High availability covers only the first row: a replica in the same cluster goes down with the cluster. And the last row is a decision, not an oversight - it is discussed at the end.

Two Instances in Production

Production’s database runs a primary and one streaming replica - instances: 2 in its Cluster, next to the archive’s serverName (below):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: app-postgres
  namespace: production
  annotations:
    cnpg.io/validation: "disabled"
    # Flux never deletes the database because it left Git (a moved folder, a profile that dropped
    # it, a bad revert): its data would go with it. Deleting it on purpose: remove it from Git first,
    # reconcile, then kubectl delete - while it is in Git, Flux re-creates it, empty (README.md).
    kustomize.toolkit.fluxcd.io/prune: disabled
spec:
  # A primary and one streaming replica (Chapter 22): when the primary's pod or disk is lost, CNPG
  # promotes the replica (24 s after a crash on kind), and an update restarts the replica first, then
  # switches over. Replication is asynchronous: transactions the replica had not received yet are
  # lost with the primary. On Hetzner both run on the one worker - this survives a lost pod, not a
  # lost node.
  instances: 2
  # Pinned to a minor and its digest: the major tag (:17) moved to every new PostgreSQL minor by
  # itself. Changing this restarts the replica, then switches the primary over to it.
  # The minimal Debian trixie image: the in-tree barman tools of the old "system" image are not needed
  # with the Barman Cloud plugin (its sidecar brings them).
  # A new MAJOR version here is an offline in-place upgrade: CNPG stops the instance and runs
  # pg_upgrade --link in a job - old and new image on the same distribution. Take a backup first; give
  # the archive a new serverName below in the same change (point-in-time restore does not cross major
  # versions); take a base backup right after (Chapter 12).
  # renovate: datasource=docker depName=ghcr.io/cloudnative-pg/postgresql
  imageName: ghcr.io/cloudnative-pg/postgresql:18.6-minimal-trixie@sha256:37ade18dbdddba430858c72725aceeec66f33fa5333e82ea1df4942f6c1c83a3
  primaryUpdateStrategy: unsupervised
  bootstrap:
    initdb:
      database: app
      owner: app
      secret:
        name: app-postgres-app
  storage:
    size: 30Gi
  # Base backups and the WAL archive go through the Barman Cloud plugin to the ObjectStore
  # app-postgres-backups (object-store.yaml); retention lives there too.
  plugins:
    - name: barman-cloud.cloudnative-pg.io
      isWALArchiver: true
      parameters:
        barmanObjectName: app-postgres-backups
        # One archive per major version AND per cluster: 17's backups and WAL stay under app-postgres
        # and still restore into a 17 cluster; 18 starts its own. BACKUP_CLUSTER_ID is the day and time
        # Terraform created this Kubernetes cluster (ConfigMap backup-s3): a rebuilt cluster starts an
        # empty archive next to the old one - the plugin refuses to archive into a used one ("Expected
        # empty archive") and the database runs without backups. The old archive stays for a restore
        # (Chapter 22); retention prunes only the live cluster's own.
        serverName: app-postgres-pg18-${BACKUP_CLUSTER_ID}
  resources:
    requests:
      cpu: 10m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 512Mi
  # initdb sets the owner's password only once. managed.roles keeps the app role's password equal to
  # the app-postgres-app Secret on every reconcile, so rotating the Secret (SOPS or Terraform) rotates
  # the database password too. The backend reads POSTGRES_PASSWORD as an env var at start: after a
  # rotation, roll it through Git (safeops.io/secrets-rotated in flux/apps/backend/production) or new
  # connections fail - not kubectl rollout restart, which Flux undoes (docs/credential-rotation.md).
  managed:
    roles:
      - name: app
        ensure: present
        login: true
        passwordSecret:
          name: app-postgres-app

• Streaming, asynchronous. The primary sends every change to the replica as it happens (pg_stat_replication: streaming, async). A commit does not wait for the replica: writes cost nothing extra. The price is that a commit the replica had not received yet is lost when the primary is. Synchronous replication closes that gap, but every write then waits for the replica, and a missing replica can stop writes. For this platform, losing what the replica had not received yet - not measured, and with no upper bound if the replica falls behind - is the better trade than making every write wait.
• The replica needs its own path. It clones the primary and then streams from it on port 5432 - and default-deny-all (Chapter 06) drops that silently. The NetworkPolicy allow-postgres-replication lets the instances of app-postgres, and only them, reach each other.
• It costs a second instance’s resources. The second instance and its Barman sidecar add 192Mi of memory requests and 768Mi of limits to production’s worst case, so production’s quota grew to 1.5Gi of requests and 4Gi of limits (Chapter 08’s check found it).
• Backups come from the replica. A backup reads the whole database; CloudNativePG takes it from the replica when there is one, so the primary’s disk and CPU stay with the application.
• Updates cost a switchover, not a restart. A new PostgreSQL minor restarts the replica first, then switches the primary over to it - about ten seconds of failed queries (below) instead of a single instance’s restart - then restarts the old primary as a replica.
• Both on one worker. On Hetzner both instances run on the only worker. This survives a crashed pod or a broken disk, not a lost node: Chapter 20’s rule, “two replicas are not two nodes”, is just as true for databases.

develop and staging stay at one instance: losing their data costs a restore, not users.

Switchover and Failover

A switchover is planned: the primary hands over to the replica on purpose - before maintenance, or to test the path. A failover is not: the primary is gone, and CloudNativePG promotes the replica.

REPLICA=$(kubectl --context kind-sre-control-plane -n production get pods \
  -l cnpg.io/cluster=app-postgres,cnpg.io/instanceRole=replica -o jsonpath='{.items[0].metadata.name}')
kubectl --context kind-sre-control-plane -n production patch cluster app-postgres \
  --subresource=status --type=merge -p "{\"status\":{\"targetPrimary\":\"$REPLICA\"}}"

A switchover sets status.targetPrimary, and the operator does the rest - the same field CloudNativePG’s kubectl cnpg promote plugin sets. It is one of the few changes made outside Git: a status, not the spec - Flux neither owns nor resets it.

Measured on kind, with a probe signing in to production’s backend four times a second (each login is one database query - a read; it shows when the database answers, not when writes come back):

The graceful delete is the surprising row. PostgreSQL’s smart shutdown waits for every client to disconnect - up to smartShutdownTimeout, 180 seconds - and serves them meanwhile; only then does the replica take over. Draining a node, or deleting a pod by hand, is that path. A crash skips it: the replica is promoted only after the operator sees the primary gone. A forced delete only removes the Pod object without waiting; on a healthy node the containers stop at once, but on an unreachable node nothing proves the old primary has stopped - two primaries are the worse outcome, so there the old one is stopped or cut off first.

The application’s side of a failover is reconnecting. The backend’s database pool drops the dead connection and opens a new one to app-postgres-rw - a Service that always points at the current primary.

One Archive per Cluster

The story’s fix, in three parts.

The archive’s name carries the cluster’s id. Terraform records when it created the cluster - built in, no extra provider - and writes it into the ConfigMap Flux substitutes into the database manifests:

resource "terraform_data" "cluster_id" {
  input = formatdate("YYYYMMDD-hhmm", timestamp())
  # A replaced kind cluster (a new node image) is a new cluster: a new id.
  triggers_replace = kind_cluster.sre.id

  lifecycle {
    ignore_changes = [input]
  }
}

timestamp() changes on every plan; ignore_changes keeps the first value until the resource is replaced - with the cluster on kind, with a new certificate authority on Hetzner. The Cluster’s plugin parameters use it - serverName: app-postgres-pg18-${BACKUP_CLUSTER_ID} in the file above.

Every new cluster writes .../app-postgres/app-postgres-pg18-<its id>/, next to its predecessors. pg18 stays in the name: a new PostgreSQL major version starts a new archive too, since point-in-time recovery does not cross major versions (Chapter 12).

The first backup at once. immediate: true on the ScheduledBackup takes a base backup when the cluster is created. Without it, a new archive holds WAL with nothing to replay it onto until the night.

Old archives are deleted by hand. Retention (retentionPolicy on the ObjectStore: 7, 14 and 30 days) prunes only the archive the live cluster writes. A predecessor’s archive stays until someone deletes it. That is the point: right after a disaster it is the only copy. Delete it once the new cluster has its own backups and nothing needs the old data any more.

Two shortcuts the story rejected, again: cleaning the path on destroy deletes the backups when a lost cluster needs them, and turning off the empty-archive check mixes two databases’ histories under one name.

The Cluster Is Lost

The order matters. The database’s bootstrap is read once, when the Cluster is created. If Flux creates it first, with initdb, production starts empty, and changing the manifest later does nothing. So the recovery goes into Git before the new cluster exists:

1. Decide it is lost. A cluster that can come back - a control plane rebooting, a provider outage - is waited for, not rebuilt. Rebuilding while the old one may return gives you two clusters writing to two archives.

2. Find the archive to restore - prove it, do not guess it. The old cluster’s server name, app-postgres-pg18-<old id>, with the id it ran with: BACKUP_CLUSTER_ID from its records (the ConfigMap flux-system/backup-s3, its Terraform run, your notes). The newest directory in the bucket is not proof - it may be a test cycle’s. The archive must hold a base backup and WAL after it; the last archived segment is the restore point. If the lost cluster never completed a backup, its data is lost: say so - restoring another cycle’s archive instead would bring back someone else’s data.

3. Declare the recovery in Git - production’s Cluster bootstraps from that archive:

   Show the recovery in production's cluster.yaml

   bootstrap:
     recovery:
       source: lost-cluster
       database: app
       owner: app
       secret:
         name: app-postgres-app
   externalClusters:
     - name: lost-cluster
       plugin:
         name: barman-cloud.cloudnative-pg.io
         parameters:
           barmanObjectName: app-postgres-backups
           serverName: app-postgres-pg18-<old id>
   

   Merged into main, like any change (Chapter 05).

4. Build the cluster. Terraform creates it - with a new id - and Flux applies main. Production’s database restores the base backup, replays the WAL up to the end of the archive, and starts. Its own archive is new and empty, and immediate: true gives it a base backup at once.

5. Prove it - three checks, not “Ready”:

   • the data: a row written after the last base backup is there - it came from the WAL, not only the backup;
   • the application: a sign-in with a password set before the loss works;
   • the new archive: ContinuousArchiving=True, and a completed backup in it.

6. Put initdb back. The recovery has done its job. Left in Git, it would restore this old archive again on the next new cluster. Removing it changes nothing on the running Cluster: CloudNativePG reads bootstrap only at creation.

Measured on kind: about 8 minutes from make kind-apply to a backend that answers - most of it the cluster’s creation and Flux applying the platform. That is the build and the restore, not the whole outage: the decision, the recovery pull request and the proofs come on top, and the outage ends with a sign-in that works. The database’s own restore was the short part. develop and staging come back empty (initdb). Their data is test data, and a recovery for them would be the same three steps.

The credentials come first. A new cluster needs the SOPS key to decrypt the Secrets in Git, a token to read the repository, and the backup store’s keys. The credential registry (Chapter 21) says where each one lives - all outside the cluster, which is why the rebuild can start at all.

etcd

A Kubernetes cluster keeps its objects in etcd, and k3s on the Hetzner cluster snapshots it into the same backup bucket (k3s’s defaults: every 12 hours, five kept). This platform does not restore from those snapshots after losing the cluster, and this is deliberate. Every object worth keeping is in Git, and Flux re-creates it on a new cluster in minutes. An etcd snapshot would bring back the objects, not the data on the volumes, and it would bring them back as they were, possibly hours old, in a cluster whose nodes no longer exist.

An etcd restore is the tool for a cluster whose objects are not all in Git - things created by hand or by controllers that keep their state in objects - and for a control plane that lost its database while the nodes live on. On k3s it is k3s server --cluster-reset --cluster-reset-restore-path=<snapshot> on the control plane. That path has not been rehearsed on this platform: until it is, it is a documented option, not a plan.

The Same Provider

The backups live with the same provider as the cluster: Hetzner Object Storage on the platform, your own machine for kind. A lost cluster is covered. A provider-wide outage makes the backups unavailable with the cluster - waiting brings both back. A lost account, or a deleted bucket, takes the backups with the cluster for good. For a course platform whose data is test data, that is accepted. For real data it is not, and the usual answers are:

• a copy in a second provider - the bucket replicated, or a second backup target in another cloud (Cloudflare R2, S3), on its own credentials;
• a replica cluster in another region (CloudNativePG) - a standby database that keeps its own archive there;
• immutable backups - object lock, so a compromised credential cannot delete them (Chapter 12’s “not immutable”).

Each one adds a credential to rotate, a cost and a restore to rehearse. Add it when the data is worth it, not before.

What Watches This

• PostgresWALArchivingFailing - the newest segment failed for 15 minutes: the story’s signal.
• PostgresBackupFailed, PostgresBackupTooOld (no base backup for 26 hours) and PostgresNeverBackedUp.
• After every new cluster: the runbook’s first check - ContinuousArchiving=True in every environment. An alert fires after 15 minutes; the check after a rebuild finds it in seconds.

What This Does Not Solve

• A lost node on Hetzner. Both instances run on the one worker. Surviving a node needs a second worker and the instances spread over both - a cost decision (Chapter 20).
• The provider or the account - above: accepted for test data.
• A mistake in the data. A replica copies a DROP TABLE within moments. The answer is a point-in-time restore to the moment before (Chapter 12), not failover.
• An etcd restore - documented, not rehearsed.
• The Hetzner numbers. The recovery times above are kind’s. On Hetzner the cluster’s creation, and pulling every image again, add to it; it is measured on the next cycle, together with the check that the platform’s earlier clusters really ran without backups.

Instructions for Your AI Agent

The rules this chapter adds:

1. A new cluster is done when every database reports ContinuousArchiving=True and has a completed backup - not when it is Ready. Enforced by: PostgresWALArchivingFailing, PostgresNeverBackedUp; the runbook’s first check.
2. Never clean, reuse or skip the check on a backup archive; a new cluster writes a new one (serverName with BACKUP_CLUSTER_ID). Enforced by: the Barman Cloud plugin’s empty-archive check; instruction.
3. Never rebuild a cluster that may still come back, and declare a database recovery in Git before the new cluster is created. Enforced by: instruction; CloudNativePG reads bootstrap only at creation.
4. Never delete an old archive while it is the newest copy of that data. Enforced by: instruction; retention never touches an archive the live cluster does not write.
5. Report failover and recovery times only from a measurement, with how it was measured. Enforced by: instruction.

This builds on: Backup & Restore (Chapter 12) - the archive, the base backup and a proven restore - Platform Lifecycle (Chapter 20), what a single worker costs, and Credential & Access Lifecycle (Chapter 21) - every credential a rebuild needs, outside the cluster. This enables: the Capstone (Chapter 23) - the whole platform, changed, proven and handed over.