Investigation

Treat maintenance state and blocking events as evidence, not as an inconvenience.

Safe investigation sequence:

1. Inspect current state: Check the current replica count and current HPA scaling status.
2. Confirm PDB allowance: Check kubectl get pdb to see how many “allowed disruptions” are remaining.
3. Compare settings: Compare the planned disruption (e.g., draining a node) with the service’s actual tolerance.
4. Identify the conflict: Determine if the PDB is blocking the drain because the service is already at its minimum replica count.

Containment

Containment is about protecting availability while resolving maintenance blockers.

Containment steps:

1. Pause the disruption: Stop the node drain if it is stalling and impacting other workloads.
2. Adjust safely: Increase minReplicas or relax the PDB safely (after peer review) rather than disabling the guardrail blindly.
3. Verify health: Ensure the service returns to a healthy multi-replica baseline before resuming maintenance.
4. Re-run correctly: Resume the maintenance step only after the allowed disruptions are clear and positive.

Pause and Predict: What automated guardrail would have prevented this incident entirely?