Incident Hook
An urgent fix is rebuilt from a developer’s workstation and pushed with a familiar image tag. The deployment appears normal, but during triage, the team cannot prove which workflow produced the binary. Signer identity, SBOM lineage, and dependency baselines are all missing.
Result: Rollback confidence drops because the trust lineage of the running artifact is uncertain. Recovery delay shifts from a technical problem to a trust problem.
Observed Symptoms
What the team sees first:
- The workload is running, but artifact provenance is unclear.
- Tags look familiar, but signature and SBOM evidence do not line up.
- Responders must investigate the identity of the code before they can trust the rollback path.
When an image “looks right,” teams are tempted to skip provenance. That is exactly when high-impact mistakes occur.
Supply Chain Baseline
To prevent untrusted code from reaching production, we enforce a strict verifiable artifact model:
- Immutable Identity: All artifacts are referenced by digest or immutable version tags.
- SBOM Generation: A Software Bill of Materials (SBOM) is generated for every release artifact.
- Signing & Attestation: Every image is cryptographically signed and attested by an authorized CI workflow.
- Admission Verification: The cluster verifies the signature and attestation before allowing the container to run.
What AI Would Propose (Brave Junior):
- “Rebuild locally and push now to save time.”
- “Skip signing for this release only.”
- “Use mutable tags for faster retagging.”
Pause and Predict: Before reading the investigation, write down your top 3 hypotheses. What would you check first?