Lab & Completion

What You Will Produce

On your own kind cluster you reproduce the incident from this chapter: two copies of the same Terraform code, each with its own local state, create the same objects twice - one with an error, one silently. You move the state to a shared backend and watch both copies agree, and a lock stop the second writer. Then you watch a reviewed plan silently undo a change made by hand, a fresh plan expose it, and the guard refuse an old plan - and you return the cluster to where it started.

Before You Start

The baseline from Chapter 00 must be clean:

flux --context kind-sre-control-plane get kustomizations -A          # every row READY=True and SUSPENDED=False
kubectl --context kind-sre-control-plane get pods -A --no-headers | grep -vE 'Running|Completed'   # prints nothing
make smoke-test                     # 9/9

Part B uses three terminals: one for most commands, one for a MinIO port-forward, one for a second Terraform run.

The Lab in Four Parts

  • A. Verify - no drift, the guard refuses to apply without a plan, a plan of unchanged code is empty, and a “sensitive” password sits in the state and the plan in plain text.
  • B. Two laptops, then one memory - two local states create the same objects twice; a shared state in MinIO gives them one memory; its lock stops a second writer.
  • C. Apply only what you have seen - a reviewed plan overwrites a hand change without a warning; a fresh plan shows it; the guard refuses an old plan; the hand change goes into code.
  • D. Rollback & cleanup - the lab objects are removed through the same workflows, the plan files are deleted, and the baseline is proven.

Open the lab →

Done When

  • terraform -chdir=infra/terraform/kind_cluster plan -detailed-exitcode returned 0 before you started (A2)
  • scripts/guard-terraform-plan.sh apply without a plan printed missing plan file (A3)
  • state show printed (sensitive value), while grep found the real database password in the state and in the saved plan (both counts above 0); infra/terraform/kind_cluster is drwx------ (A5)
  • laptop B’s apply printed configmaps "lab-migration-db" already exists and created a second lab-worker-...; the two state lists differed (B3-B4)
  • after both copies used the MinIO backend, laptop B’s plan returned exit=0 (B8)
  • while laptop A waited at Enter a value:, laptop B got Error acquiring the state lock (B9)
  • after applying the reviewed plan, cluster-config said v2, not hotfix-by-hand (C4)
  • a fresh plan showed "hotfix-by-hand" -> "v2" (C6), and the guard printed plan is too old (120m > 60m) (C7)
  • with the hand change in code, plan -detailed-exitcode returned 0 (C8)
  • at the end, no lab- ConfigMaps are left, plan -detailed-exitcode returns 0, git status --short prints nothing, and the three baseline commands are clean (D1-D4)

Knowledge Check

Before finishing this chapter, complete the Quiz.