Investigation & Containment

Investigation

Five questions, in this order: is the database backed up right now, where does it write, can that archive be restored, what survives losing the cluster - and what does one lost instance cost. Every kubectl names the cluster with --context (Chapter 01).

Is it backed up right now? Not “is it Ready” - archiving is its own condition:
```
kubectl --context kind-sre-control-plane -n production get cluster app-postgres \
  -o jsonpath='{range .status.conditions[?(@.type=="ContinuousArchiving")]}{.type}={.status} {.reason}{"\n"}{end}'
```
ContinuousArchiving=True ContinuousArchivingSuccess - every WAL segment reaches the archive. In the story it read False ContinuousArchivingFailing on a cluster that was otherwise healthy; the plugin’s log, in the database pod’s plugin-barman-cloud container, said why: Expected empty archive.
Where does it write? The ObjectStore holds the bucket and the path; the Cluster adds the server name - one directory per archive:
```
kubectl --context kind-sre-control-plane -n production get objectstore app-postgres-backups -o jsonpath='{.spec.configuration.destinationPath}{"\n"}'
kubectl --context kind-sre-control-plane -n production get cluster app-postgres -o jsonpath='{.spec.plugins[0].parameters.serverName}{"\n"}'
```
s3://sre/cnpg-backups/production/app-postgres and app-postgres-pg18-20261012-1950. In the story the server name was app-postgres-pg18 on every cluster: a new database found the old one’s history in its directory, and the plugin refused to mix them. The suffix now is the day and time Terraform created the cluster - BACKUP_CLUSTER_ID in the ConfigMap flux-system/backup-s3 - so every new cluster starts an empty archive of its own, next to the old ones.
Can that archive be restored? WAL alone restores nothing; it needs a base backup to replay onto. The ObjectStore’s status keeps the window per archive:
```
kubectl --context kind-sre-control-plane -n production get backups
kubectl --context kind-sre-control-plane -n production get objectstore app-postgres-backups -o jsonpath='{.status.serverRecoveryWindow}{"\n"}'
```
One completed backup, taken about a minute after the database started, and a recovery window from firstRecoverabilityPoint - a restore can reach any moment from there to the last archived WAL. In the story a new cluster’s first backup waited for the night; the ScheduledBackup now has immediate: true.

What survives losing the cluster? List what the platform needs to run, and where each lives:

What	Where it lives	Survives the cluster?
the cluster itself	Terraform code + state	yes - code in Git, state in R2 (Hetzner) or on your machine (kind)
every object Flux applies	Git	yes
the keys to decrypt SOPS and push to Git	the credential registry’s `lives_in` (Chapter 21)	yes - outside the cluster by design
the databases’ data	their volumes	no
the backups	object storage: Hetzner Object Storage; on kind, MinIO’s data directory on your machine	yes - if it is outside the cluster

One row does not survive, and it is the one Git cannot rebuild. Everything else is code; the data is the reason backups exist. On kind, MinIO runs inside the cluster - but its data is a directory on your machine (infra/terraform/kind_cluster/.minio-data), mounted into the node, so make kind-destroy leaves it.

What does one lost instance cost? Chapter 20 measured it: every database went down with the only worker, and came back by itself about a minute and a half after the reboot. A single instance turns every lost pod, disk or node into an outage of at least that long - and a lost disk into a restore.

The root cause is not the plugin’s check, which was right. It is one archive path shared by every cluster that ever existed, and a health signal that did not include backups - so a rebuilt cluster could run healthy and unprotected, and nothing in the platform’s checks noticed.

Containment

A new archive for every new cluster. Terraform writes the cluster’s id into backup-s3; the server name carries it. A rebuilt cluster archives from its first minute, and the old archive stays untouched - it is what a restore needs.
The first backup at once. immediate: true on every ScheduledBackup: a new archive gets its base backup when the cluster is created, not the night after.
Check archiving after every new cluster - ContinuousArchiving=True in all three environments, before the cluster counts as done (the runbook’s first step).
Two instances in production. A primary and a streaming replica: a lost pod or disk is a promotion in seconds, not an outage or a restore.
Rehearse the loss. The plan “rebuild from code, restore from backup” is only proven by doing it - on kind in this chapter’s lab, end to end.

A backup is proven by a restore - and a recovery plan by losing the cluster on purpose.

Pause and Predict: The cluster is gone. Terraform can build a new one in minutes, and Flux applies everything in Git. Where - and when - do you say that production’s database must come from the old archive instead of starting empty?