Chapter 22: Disaster Recovery
Learning Objectives
By the end of this chapter, you will be able to:
- Say for each failure - a pod, a disk, a node, the cluster, the provider - what brings the platform back, how long that takes and how much data it can lose - and compare both with the objectives (RTO, RPO)
- Run a database as a primary and a streaming replica, and measure what a switchover and a failover cost the application
- Explain why a rebuilt cluster must not write into its predecessor’s backup archive - and how a database can run “healthy” with no backups at all
- Rebuild the platform after losing the whole cluster: the cluster from Terraform, everything else from Git, and the data from its backup - declared in Git before the cluster exists
- Say what the platform does not protect against, and what that would cost to change
Recall Check
Before continuing, quickly recall:
- What do a base backup and the WAL archive each hold, and what does a point-in-time restore need from both? (Chapter 12)
- What happens to the platform’s pods when its only worker reboots? (Chapter 20)
- Where is every credential the platform needs to start, and which one opens all the others? (Chapter 21)
If you can’t answer these, revisit the corresponding chapter before proceeding.
Context
Chapter 12 proved that a backup restores - next to a database that was still there. A disaster is the other case: the database, or the whole cluster, is gone, and the restore is the only way back. Two questions decide how bad that day is: how long until it works again and how much is lost. The answers you accept are the objectives - the recovery time objective (RTO) and the recovery point objective (RPO), as in Chapter 12; the answers you get are only known by doing it, and show whether the objectives hold.
This platform answers them in two layers:
- High availability for the small, frequent failures: production’s database runs a primary and a replica, so a lost pod or disk costs seconds, not a restore.
- Disaster recovery for the large, rare ones: when the cluster is lost, Terraform builds a new one, Flux puts back everything that is in Git - and the data comes back from its backup, declared in Git too.
This chapter shows one widely used way to do it, not every way: CloudNativePG’s streaming replica and its Barman Cloud backups, a cluster rebuilt from code. Others work too:
- synchronous replication - a commit waits until a replica has it, so losing the primary loses nothing; every write pays the round trip, and a missing replica can stop writes;
- a managed database (a cloud’s PostgreSQL service) - failover, backups and point-in-time restore are the provider’s; the platform keeps only its configuration;
- a standby cluster in another region (a CloudNativePG replica cluster, storage replication) - a second running copy of the platform: the shortest recovery time, at about twice the cost;
- cluster backups with Velero - Kubernetes objects and volumes saved from the cluster itself, for platforms whose state is not all in Git.
The story is this platform’s own: the rehearsal for this chapter found that a rebuilt cluster wrote no backups at all.
Work through each lesson section; the chapter video, once it is published, gives the concept overview.
From a Lost Cluster to a Working One
