Core Track Guardrails-first chapter in core learning path.

Estimated Time

  • Reading: 30 min
  • Lab: 35 min
  • Quiz: 5 min

Prerequisites

What You Will Produce

On your own kind cluster: the HPAs and PodDisruptionBudgets of the three environments read as numbers, proof that the HPA - not Flux - owns the replica count, a drain of the worker simulated with --dry-run=server and the PDB that would stop it named; then, in the namespace lab, an eviction refused by a budget, one allowed - and the budget gone until the replacement is Ready - and an HPA that scales a busy Deployment from one pod to three. The lab ends with lab empty, the worker schedulable and every Kustomization Ready and none suspended.

Chapter 09: Availability Engineering (HPA + PDB)

Learning Objectives

By the end of this chapter, you will be able to:

  • Explain who owns an app’s replica count - the HPA, not Git - and why a spec.replicas in Git would fight it during a peak
  • Tell what a PodDisruptionBudget guards (evictions: a drain before an upgrade or a reboot) from what it does not (rolling updates, crashes, node failures)
  • Read a budget: minAvailable against maxUnavailable, and what ALLOWED DISRUPTIONS means for two pods and for three
  • Prepare a drain with evidence - where the pods run, where their replacements could go, which budgets block - and simulate it without changing the node

Recall Check

Before continuing, quickly recall:

  • What does Flux do with a field that is in Git and was changed in the cluster? (Chapter 03)
  • The quota refused a pod of a Deployment. Where did the refusal appear? (Chapter 08)
  • What do a pod’s requests mean to the scheduler? (Chapter 08)

If you can’t answer these, revisit the corresponding chapter before proceeding.

Context

Chapter 08 gave every container a budget. This chapter keeps enough of them running - while the load changes and while the nodes underneath are maintained.

Two things take pods away. Load needs more of them than are running: the HorizontalPodAutoscaler (HPA) watches the pods’ CPU and memory and changes the Deployment’s replica count between a minimum and a maximum. Maintenance removes them on purpose: a node is drained before an upgrade or a reboot, and every pod on it is evicted. A PodDisruptionBudget (PDB) tells that eviction how many pods of an app must stay.

Both look like settings. Both are really promises - and both break in quiet ways. An HPA whose replica count is also written in Git is set back by Flux at the worst moment. A PDB stops the eviction, but it cannot make room: if the evicted pods have nowhere to go, the budget only turns an outage into a drain that never ends. And two replicas protect against a node failure only if they run on two nodes.

The platform’s answer is to count, not to hope: the HPA alone owns the replica count, every app has a budget, and every drain starts as a simulation that names what would stop it. This chapter shows each piece - and is honest about what the platform’s single worker node can and cannot promise.

Work through each lesson section; the chapter video, once it is published, gives the concept overview.

Two Ways Pods Go Away

Diagram: load; metrics-server CPU / memory per pod; HPA min ≤ replicas ≤ max; Deployment spec.replicas (owned by the HPA, not Git); ReplicaSet keeps that many pods; maintenance; kubectl drain cordon + evict; Eviction API; PDB allowed disruptions; 1 or more; pod evicted ReplicaSet creates a replacement; 0; 429: refused drain retries until its timeout

Detailed Lessons

Hands-On Materials

Labs, quizzes and runbooks for this chapter are free - sign in with your email, Google or GitHub to open them.

  • Lab: Count, Don't Hope Free · sign in
  • Quiz: Chapter 09 (Availability Engineering: HPA + PDB) Free · sign in
Sign in free