Investigation & Containment

Investigation

Treat the SRE Guardian itself as a guarded incident pipeline.

Safe investigation sequence:

1. Inspect Raw Signals: Review the raw Kubernetes events and metrics entering the Guardian.
2. Verify Sanitization: Confirm that secrets, tokens, and context budgets are correctly handled before LLM analysis.
3. Confirm Deduplication: Ensure that the Guardian correctly collapsed multiple related alerts into a single incident record.
4. Review Proposed Actions: Check if the AI-suggested actions are useful and stay within the “no-mutation” boundary.

Containment

Containment keeps the Guardian helpful but safely bounded.

Containment steps:

1. Preserve Human Approval: Do not allow any remediation step to execute without explicit human sign-off.
2. Reduce Noise: Tune deduplication and escalation rules to prevent alert fatigue.
3. Block Unsafe Context: Regularly audit the sanitization logic to prevent secret leakage.
4. Treat Low Confidence as Review: Handle incidents with low AI confidence as high-priority human review items rather than automation failures.

Pause and Predict: What automated guardrail would have prevented this incident entirely?