Investigation & Containment

Investigation

Treat canary metrics as the definitive proof of release safety.

Safe investigation sequence:

1. Verify Traffic Split: Check the current traffic distribution between the stable and canary versions.
2. Monitor Canary Metrics: Analyze the latency and error rates specifically for the canary pods.
3. Compare with Stable: Compare canary performance against the baseline of the stable version.
4. Inspect Flagger Events: Review the Flagger logs or Kubernetes events to see why an analysis is progressing or stalled.

Containment

Containment is built directly into the progressive delivery engine.

Containment steps:

1. Automatic Halt: Flagger automatically stops the traffic shift if analysis thresholds are violated.
2. Immediate Revert: Traffic is shifted back to 100% stable version instantly upon failure detection.
3. Isolate Canary Pods: Keep the failed canary pods running (but without traffic) for debugging.
4. Fix Upstream: Resolve the bug in Git and push a new version to restart the canary cycle.

The goal is “automated safety,” where the system protects the user from bad code.

Pause and Predict: What automated guardrail would have prevented this incident entirely?