Lab & Completion

What You Will Produce

On your own kind cluster: production’s database as a primary and a streaming replica, and the archive this cluster writes - its name, its first backup, its recovery window. A switchover and a crash, each timed from the application’s side by a probe. Then a row written after the last base backup, the whole cluster deleted, the recovery declared in Git, a new cluster built - and four proofs: the row is back, the schema was there before the backend, a sign-in with the old password works, and the new cluster writes an archive of its own.

Before You Start

The baseline from Chapter 00 must be clean:

flux --context kind-sre-control-plane get kustomizations -A          # every row READY=True and SUSPENDED=False
kubectl --context kind-sre-control-plane get pods -A --no-headers | grep -vE 'Running|Completed'   # prints nothing
make smoke-test                     # 9/9

You need yq, gh (signed in), curl and openssl, and about an hour - two cluster builds of seven to ten minutes each.

Your fork and your cluster must be newer than this chapter. Open your fork on GitHub, click Sync fork → Update branch, then git switch main && git pull. If infra/terraform/kind_cluster/.minio-data does not exist after that, your cluster still keeps MinIO’s data inside itself - rebuild it once:

make kind-destroy
make kind-plan && make kind-apply

The Lab in Three Parts

  • A. Verify - two instances, and the archive this cluster writes.
  • B. Break & Fix - a switchover and a crash, timed; a row after the last backup; the cluster deleted and brought back through Git.
  • C. Rollback & Cleanup - initdb back in Git, the baseline.

Open the lab →

Done When

  • you can say which instance is the primary, how the replica streams, and what the budget protects (A2)
  • you can name your cluster’s archive, its first backup and its recovery window (A3)
  • your predictions were written before B3 (B1)
  • you read the switchover’s and the crash’s cost from the probe, and can say why the crash costs more (B3-B4)
  • a row existed that only the WAL after the last base backup holds, and the segment holding it was in the store (B5)
  • before the cluster was deleted: the archive’s name, a base backup in it and B5’s segment - then deleted, and the archive still there (B6)
  • on the new cluster: the row, the schema, a sign-in with the old password, and a new archive with its own first backup (B7)
  • initdb is back in Git, the database untouched, and the baseline is clean (C1-C2)