Chapter 15: 24/7 Production SRE

Learning Objectives

By the end of this chapter, you will be able to:

• Assign incident roles: Commander, Responder, Comms Lead, Scribe
• Apply the 4-tier severity matrix for triage decisions
• Conduct a blameless postmortem focused on system gaps, not individual blame
• Convert incident findings into concrete hardening tasks

Start with the video for the concept overview, then work through each lesson section.

Recall Check

Before continuing, quickly recall:

• What guardrails must be in place before any chaos drill? (Chapter 13)
• Why can the AI guardian enrich and route alerts but never execute mutations? (Chapter 14)
• Which three signals must agree before taking action during an incident? (Chapter 11)

If you can’t answer these, revisit the corresponding chapter before proceeding.

Tooling is useless without operational discipline. When a major incident happens at 3:00 AM, the difference between a 15-minute fix and a 4-hour outage is how your team coordinates. In this final core chapter, we implement the SRE operating model to scale beyond individual heroism.

1. The Problem: The “Coordination Chaos”

A critical alert (Sev1) fires outside of business hours. Responders join, but roles are unclear and communication is fragmented. Technical actions race each other, creating further drift and uncertainty. Time is lost on coordination instead of restoration.

2. The Concept: Incident Roles & Severity

We treat an incident like a structured mission with clear ownership:

1. Incident Commander (IC): Strategy and resource management.
2. Primary Responder: Technical execution and verification.
3. Communications Lead: Stakeholder updates and status reporting.
4. Scribe: Timeline and evidence logging.

3. The Code: Runbooks & Severity Matrix

Our sre/ repo includes the organizational templates required for high-stakes response. We use a 4-tier severity matrix to set communication expectations and response targets.

4. The Guardrail: Blameless Postmortems

The most important SRE guardrail is the Postmortem. We focus on system conditions and guardrail gaps rather than individual mistakes. Every major incident must result in specific, evidence-backed hardening actions.

5. Verification: Did I Get It?

Run an incident simulation using your Chaos Monkey and follow the Safe Workflow:

# 1. Detect signal (Chapter 11)
# 2. Declare Severity and Assign Roles
# 3. Build Timeline and Mitigate
# 4. Confirm Recovery and Resolve

Expected Output: A clear, auditable timeline of the incident and a draft postmortem with at least one technical action item.

Detailed Lessons

• The Incident: Coordination Chaos
• Investigation & Containment
• Workflow & Operating Model
• Lab & Completion
• Quiz