Investigation

Treat restore validation as the real test, not the backup status line.

Safe investigation sequence:

1. Confirm Artifact: Verify the backup artifact exists and matches the retention policy.
2. Restore Locally: Restore into an isolated, non-production target (e.g., a restore-test namespace).
3. Verify Data: Prove the application can actually use the restored data (schema, permissions, sample reads/writes).
4. Identify Gaps: Pinpoint why the restore failed (e.g., missing role grants or bootstrap scripts).

Containment

Containment is about restoring confidence in the recovery path before touching production.

Containment steps:

1. Gate Decisions: Keep production recovery decisions gated on successful non-production restore evidence.
2. Fix Gaps: Address permission, schema, or bootstrap gaps in the lower environment first.
3. Document the Path: Write down the validated restore steps once they are repeatable.
4. Promote Procedure: Only promote the recovery procedure once it is proven reliable.

Pause and Predict: What automated guardrail would have prevented this incident entirely?