Investigation & Containment

Investigation

Before a drain - or while one hangs - read where the pods are and where they could go. Every command names the cluster with --context (Chapter 01); on your kind cluster the same commands answer for your own worker, which has the same layout as the story’s: one control plane that takes no workloads, one worker that runs everything.

Which nodes can take a pod?
```
kubectl --context kind-sre-control-plane get nodes \
  -o custom-columns='NODE:.metadata.name,UNSCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints[*].key'
```
The control plane carries the taint node-role.kubernetes.io/control-plane - no app tolerates it. The worker has no taint, and UNSCHEDULABLE <none>: it is not cordoned. A drained node shows true here - and keeps it until kubectl uncordon. One schedulable node means a drain has nowhere to move anything.
What runs on the node you want to drain?
```
kubectl --context kind-sre-control-plane get pods -A --field-selector spec.nodeName=sre-control-plane-worker \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind'
```
Every workload of the cluster: apps (ReplicaSet), Postgres (Cluster - the CloudNativePG operator), DaemonSets, which a drain leaves alone. Count what has a second replica on another node: here, nothing.
Which budgets would block - in every namespace?
```
kubectl --context kind-sre-control-plane get pdb -A
```
The ALLOWED DISRUPTIONS column is the answer per app. On your cluster: app-postgres-primary with 0 in develop, staging and production - the operator’s budget for the primary - in develop and staging their only instance; the apps’ budgets allow 1. A drain is blocked by the strictest budget among all the pods on the node, not by the namespace you are thinking of.
Where do an app’s replicas run? Two replicas protect against a node failure only on two nodes:
```
kubectl --context kind-sre-control-plane -n staging get pods -l 'app in (backend,frontend)' -o wide
```
The NODE column: all four pods on sre-control-plane-worker. On this platform, two replicas protect against a crashed pod and make a rolling update seamless - not against losing the node.
Is the HPA in control?
```
kubectl --context kind-sre-control-plane -n production describe hpa backend
```
Under Conditions: AbleToScale True, ScalingActive True ValidMetricFound - it reads metrics and may scale; ScalingLimited True TooFewReplicas - the load would need fewer pods than the minimum of 2, so it stays at 2. ScalingActive False would mean no metrics: the HPA then changes nothing.
What would the drain do - without doing it?
```
kubectl --context kind-sre-control-plane drain sre-control-plane-worker \
  --dry-run=server --ignore-daemonsets --delete-emptydir-data --timeout=20s
```
--dry-run=server asks the API server to check every eviction and changes nothing - the node is not even cordoned; --timeout ends the retries. The output lists the pods it could evict and, for the ones it cannot: Cannot evict pod as it would violate the pod's disruption budget, then global timeout reached: 20s for each app-postgres-1. Because nothing is really evicted, each check sees the budgets as they are now: the dry run finds the budgets that block already - here, Postgres - but not the ones a real drain would hit later. In a real drain the first backend pod goes, its replacement stays Pending, and the budget then refuses the second - the dry run checks both pods against an unchanged budget and lets both through. That is what steps 1, 2 and 4 are for: where every replacement would run.

The root cause is not the budgets and not the drain command. It is a drain with nowhere to go: one schedulable node, and pods whose replacements could only start on the node being emptied.

Containment

Stop the drain - and remember it is not a rollback: the node stays cordoned.
Uncordon the node: kubectl --context kind-sre-control-plane uncordon sre-control-plane-worker (on your cluster) makes it schedulable again; the pending pods start.
Leave the budgets alone. They are in Git; a deleted or loosened one is a promise removed for every later drain.
Plan the maintenance as what it is: with one worker, a reboot is an outage of everything on it - announced, with Postgres stopped cleanly. To make a drain invisible, the platform first needs a second node where the replacements can run.

A PodDisruptionBudget can refuse an eviction; it cannot make room for one. Before a drain, prove where every evicted pod would go.

Pause and Predict: Flux applies the backend Deployment from Git every few minutes. If Git said replicas: 2 and the HPA had just scaled the backend to 3 during a peak - what would the next reconcile do?