Investigation

Treat coordination and communication gaps as part of the incident, not as background noise.

Safe investigation sequence:

1. Declare Severity: Explicitly name the severity and assign core roles immediately.
2. Build the Timeline: Create a shared, real-time timeline from metrics, logs, and operator actions.
3. Separate Evidence: Clearly distinguish between confirmed facts and assumptions or hypotheses.
4. Audit Coordination: Identify if parallel work is happening without the Incident Commander’s knowledge.

Containment

Containment in SRE is both organizational and technical.

Containment steps:

1. Establish Command: Assign an Incident Commander (IC) to manage the people and the strategy.
2. Set Update Cadence: Commit to a fixed communication interval based on the severity level.
3. Lowest-Risk Mitigation: Execute the simplest action that matches the available evidence.
4. Confirm Recovery: Don’t wind down the “War Room” until metrics confirm service recovery.
5. Open Follow-ups: Capture immediate action items while the context is still fresh.

Pause and Predict: What automated guardrail would have prevented this incident entirely?