Lab & Completion

What You Will Produce

On your own kind cluster, the platform’s availability settings read as numbers: the HPAs and budgets of develop, staging and production, proof that the HPA - not Flux - owns the replica count, the placement of every replica on the one schedulable worker, and the incident’s drain simulated with --dry-run=server, naming the budgets that would stop it while the node stays untouched. Then, in lab: an eviction refused by a budget, one allowed - with the budget at 0 until the replacement is Ready - and an HPA that scales a busy Deployment from one pod to three by itself. The lab ends with lab empty and the worker schedulable; no node is drained for real.

Before You Start

The baseline from Chapter 00 must be clean:

flux --context kind-sre-control-plane get kustomizations -A          # every row READY=True and SUSPENDED=False
kubectl --context kind-sre-control-plane get pods -A --no-headers | grep -vE 'Running|Completed'   # prints nothing
make smoke-test                     # 9/9

This chapter also needs labs/availability/ in your clone: if it is missing, your fork must first catch up with the course repository: on GitHub, open your fork and click Sync fork → Update branch. Then git switch main && git pull - it fetches from your clone’s remote (your fork) and merges the changes into your local main; it does not sync your fork with the course repository, which is why the Sync fork step comes first.

The Lab in Three Parts

  • A. Verify - the HPAs and budgets; who owns spec.replicas; the nodes and where the replicas run; the drain simulated.
  • B. Break & Fix - the Eviction API against a strict budget and a budget of one; an HPA scaling on its own.
  • C. Rollback & Cleanup - the lab’s objects deleted, lab empty, the worker still schedulable, the baseline proven.

Open the lab →

Done When

  • get hpa -A and get pdb -A showed the three environments’ numbers and app-postgres-primary at ALLOWED DISRUPTIONS 0 (A2)
  • the staging backend’s replica count is the HPA’s: Git sets no spec.replicas, and no field manager owns it - so Flux never sets it back (A3)
  • all staging replicas ran on sre-control-plane-worker, the only node without a taint (A4)
  • the drain’s dry run named Cannot evict pod as it would violate the pod's disruption budget for Postgres, and the worker stayed unschedulable= (A5)
  • the eviction against minAvailable: 2 was refused with TooManyRequests, and against minAvailable: 1 accepted - with the budget at 0 until the replacement was Ready, then 1 again (B1-B2)
  • the HPA scaled busy to 3, with ScalingLimited True TooManyReplicas (B3)
  • at the end kubectl --context kind-sre-control-plane -n lab get pods,pdb,hpa prints No resources found, the worker is schedulable, and the baseline is clean (C1-C2)

Knowledge Check

Before finishing this chapter, complete the Quiz.