Investigation
Start by proving identity, not by rebuilding again.
Safe investigation sequence:
- Compare digests: Compare staging and production image digests directly.
- Inspect Git evidence: Review the promotion commit and metadata in the repository.
- Confirm Flux actions: Confirm what
ImagePolicyselected and whatImageUpdateAutomationwrote back to Git. - Determine lineage: Decide whether the deployment was a real promotion or a new build wearing a familiar name.
Containment
Containment restores one trustworthy artifact path.
Containment steps:
- Revert to known-good: Revert to the last known-good production promotion commit in Git.
- Flux Reconcile: Let Flux reconcile the previous, stable digest.
- Verify identity: Verify that the deployed workload matches the intended immutable artifact.
- Re-run correctly: Re-run the promotion only after the artifact lineage is clear again.
Pause and Predict: What automated guardrail would have prevented this incident entirely?