The Incident: Ready, and No Backups

Incident Hook

This one happened on this platform, while this chapter was being prepared.

The platform’s databases had backups since Chapter 12: a base backup every night, the WAL archived continuously, four alerts watching it, and a restore proven next to the original. Every database ran one instance - in every environment.

Chapter 20 had already shown what one instance costs: when the only worker rebooted for its updates, every database went down with it and came back by itself about a minute and a half later. That was a planned outage, in a window.

The plan for this chapter was a rehearsal of the unplanned one: lose the whole cluster, and bring production back from its backup. The platform is built from code - Terraform for the cluster, Flux for everything in Git - and the Hetzner cluster is rebuilt for every test cycle anyway. So the first question was simple: when a new cluster starts, what does its database do with the archive the old one left behind?

The answer came from a test cluster on kind, pointed at the archive path a new cluster would use (2026-10-12):

ERROR: WAL archive check failed for server app-postgres-pg18: Expected empty archive

And the cluster’s status:

Ready=True                ClusterIsReady
ContinuousArchiving=False ContinuousArchivingFailing

The new database was Ready. The application could use it. And nothing it wrote was backed up: the backup plugin refuses to archive into a path that already holds another database’s history - rightly, since mixing two histories would make both unrestorable - and the database runs on without it. The only signal was an alert, PostgresWALArchivingFailing, fifteen minutes later - on kind, where platform alerts reach nobody.

The archive path was the same for every cluster: cnpg-backups/<environment>/app-postgres/app-postgres-pg18. On kind, until this chapter, the backups lived inside the cluster and disappeared with it, so every new kind cluster started with an empty path. (Now MinIO’s data is a directory on your machine and survives make kind-destroy - which is exactly what made the problem appear on kind too.) On Hetzner they live in object storage, which outlives every cycle - and nothing cleaned that path. So every Hetzner cluster after the first most likely ran exactly like the test: Ready, smoke tests green, and no backups. The next Hetzner cycle checks it.

The rehearsal found a second gap right behind the first: a new cluster took its first base backup at the next scheduled time, during the night. Until then the archive held WAL with no base backup to replay it onto - nothing in it could be restored.

Result: no data was lost - no disaster happened. But the plan “rebuild from code, restore from backup” would have failed at its last step on the platform where it mattered: the backups it needed were never written.

Observed Symptoms

  • A database Ready and serving, its cluster Cluster in healthy state - and ContinuousArchiving=False in its conditions.
  • The plugin’s log: Expected empty archive, once per WAL segment.
  • On a new cluster: no base backup until the night after it was created.
  • Every database a single instance: any lost pod, disk or node is an outage until it comes back.

Confusion Phase

The first readings were reassuring:

  • “The cluster says it is healthy - so backups are fine.”
  • “Backups are configured in Git, and Git is the same on every cluster.”
  • “We restored in Chapter 12, so restore works.”

Each was true of something else. Healthy means the database serves - archiving is a separate condition. The configuration was the same, and that was the problem: the same path, on a new database. And Chapter 12’s restore proved an old cluster’s backups, next to it; it said nothing about a new cluster’s.

What AI Would Propose (Brave Junior)

  • “Clean the archive path when the cluster is destroyed - then a new cluster starts empty.”
  • “Turn off the archive check - skipEmptyWalArchiveCheck - and the error goes away.”
  • “Run three database replicas in production, then we never need a restore.”
  • “Back up the cluster’s etcd every hour - then we can restore everything.”

Why This Is Dangerous

  • Cleaning the archive on destroy deletes the backups exactly when a disaster needs them. A lost cluster is a destroyed cluster. The archive has to outlive it - which means a new cluster must write somewhere new.
  • Turning off the check mixes two histories in one archive. The new database’s WAL lands next to the old one’s, under the same name; a restore can no longer tell which timeline it is replaying. The check is right; the path is wrong.
  • Replicas are not backups. A replica receives every change within moments - including a DROP TABLE, a bad migration, or a row deleted by mistake. High availability shortens the small outages; it does not undo a mistake, and it is in the same cluster when the cluster is lost.
  • An etcd backup restores the cluster’s objects, not its data. The databases’ volumes are not in etcd. On a platform where every object comes from Git, the cluster’s state is rebuilt by Flux; what has to be restored is the data.
  • Healthy is not backed up. A check that looks only at “Ready” says nothing about the one property that matters on the worst day.

Pause and Predict: A new cluster’s database is Ready, the application works, and the smoke tests pass. What one field in the database’s status would tell you whether anything it writes today can be restored tomorrow?