The Incident: The Drain That Would Not Finish

Incident Hook

Saturday, 09:00. Boris has a maintenance window on the Hetzner cluster: the worker node needs a reboot after a kernel update. With the cluster’s admin kubeconfig - the one maintenance needs - he runs what every runbook says:

kubectl drain sre-worker --ignore-daemonsets --delete-emptydir-data

The node is cordoned. Pods start to leave: Traefik, Grafana, one backend and one frontend pod in each environment… Then the output stops moving:

error when evicting pods/"app-postgres-1" -n "production" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"backend-7d4cdcd4c5-lwbg7" -n "production" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Every five seconds, the same lines - for Postgres and for the second pod of every app. Ten minutes later, still.

09:15. Boris asks his AI agent. It knows the flags: add --disable-eviction - it deletes the pods instead of evicting them and skips the budgets - or delete the PDBs for the duration of the window.

09:20. Ana, on call, asks one question before anything else: where would those pods go? The answer is on one screen:

NAME                           STATUS                     ROLES
sre-control-plane              Ready                      control-plane
sre-worker                     Ready,SchedulingDisabled   <none>

One worker - and it is the one being drained. The control plane takes no workloads. Every pod that “left” is Pending: 0/2 nodes are available: 1 node(s) had untolerated taint ..., 1 node(s) were unschedulable. Among them Traefik - one replica, no budget - so the site is already down. The budgets held the rest: one pod of each app still runs on the cordoned node, and Postgres - the one pod in each namespace whose data has no second copy - was never touched.

09:25. Boris stops the drain with Ctrl-C. Nothing comes back: the node is still cordoned, and the pending pods stay pending. Ana runs kubectl uncordon sre-worker; the pending pods start again within a minute, Traefik among them.

Monday: the reboot is planned as what it is with one worker - a short outage of everything, announced, with Postgres stopped cleanly - and a second worker goes into the Terraform backlog.

Result: thirty minutes of a dead site in a maintenance window that was meant to be invisible, and two “fixes” that would have stopped the databases too - stopped by a budget that could not make room, only refuse.

Diagram: 09:00 drain the worker pods evicted; Postgres, 2nd app pods: PDB allowed disruptions 0 retry every 5s; evicted pods Pending no other node Traefik gone: site down; 09:15 --disable-eviction or delete the PDBs; 09:20 where would they go? nowhere; 09:25 stop the drain uncordon

The story is invented; the single worker, the budgets and the messages are this platform’s - your kind cluster has the same layout, and the lab simulates this drain on it.

Observed Symptoms

  • kubectl drain does not finish: Cannot evict pod as it would violate the pod's disruption budget for Postgres and for the second pod of each app, again every five seconds.
  • kubectl get nodes: the worker Ready,SchedulingDisabled - cordoned.
  • The evicted pods are Pending: node(s) were unschedulable and an untolerated taint on the control plane. Traefik is one of them - the site is down.
  • After the drain is stopped: the node stays cordoned; nothing returns by itself.

Confusion Phase

It looks like the budgets are the problem:

  • the drain worked for every pod except the ones with a strict budget;
  • the runbook’s command is the standard one;
  • the flags to get past the budgets exist - so they must be meant for this.

Is the PDB blocking the maintenance - or is it the last thing telling the truth about it?

What AI Would Propose (Brave Junior)

  • “Add --disable-eviction - it deletes the pods directly and skips the budgets.”
  • “Delete the PDBs for the window and recreate them afterwards.”
  • “Skip the drain and just reboot the node - the pods come back with it anyway.”

Why This Is Dangerous

  • A budget cannot make room. It decides whether a pod may be evicted; it cannot create a node for the replacement. With one worker, every evicted pod is a pod that does not come back until the node does - and a pod with no budget, like Traefik here, goes first.
  • --disable-eviction deletes instead of evicting - the budgets are never asked. The pods still get their normal shutdown, but Postgres would have been stopped in all three environments, with nowhere to start again: exactly what its budget was there to refuse.
  • A deleted PDB is a promise removed for every future drain, not only for this one - and Flux puts it back from Git later, after the damage.
  • A reboot without a drain asks no budget at all. Only the Eviction API consults a PDB; a node that goes down takes its pods with it - the budgets guard the drain before a reboot, not the reboot.
  • A stopped drain leaves the node cordoned. Stopping it is not a rollback; uncordon is.
  • The evidence was one command away: the nodes, their taints, and where each pod would go.

Pause and Predict: In staging, the backend has two replicas and a budget of minAvailable: 1. During a drain, how many of the two may be evicted at once - and what happens to the second eviction while the first pod’s replacement is still starting?