Incident Hook

A developer pushes directly to main, skipping validation. An unreviewed application or infrastructure change reaches production. Because the change was never reviewed or planned, the team has no evidence of the intent behind the mutation.

Result: An untested change breaks production, and responders must reconstruct the intent after the damage has already landed.

Observed Symptoms

What the team sees first:

• There is no pull request discussion for the change.
• No approved plan artifact exists for the infrastructure mutation.

The “Invisible Change” (CI Bypass):

# On a local terminal skipping all hooks:
git commit -m "fix: change cpu limits" --no-verify
git push origin main # ❌ Bypass: PR was never opened, AI-review never ran.

# Result in cluster: 
# Pods are restarting with new limits, but NO ONE knows why or who did it.

The first operational problem is missing evidence, not just missing tooling.

Confusion Phase

The workflow now feels partially intact because some automation still ran. That creates false confidence. The real question becomes:

• Which guardrail failed first?
• Which missing layer allowed the later layers to become insufficient?

Guardrails Layering Model (Defense in Depth)

We use a layered defense model where each layer assumes the previous one might have been bypassed:

1. Local (pre-commit): Catches mistakes immediately on the developer workstation.
2. CI (GitHub Actions): Enforces validation and planning in a clean, isolated environment.
3. Review (CodeRabbit AI): Provides an automated safety net to catch human patterns.
4. Approval Gate: The final human checkpoint before any production apply.

What AI Would Propose (Brave Junior):

• “Skip pre-commit hooks locally, CI will catch it.”
• “Apply Terraform directly, we already know what it does.”
• “Merge without review, the change is small.”

Pause and Predict: Before reading the investigation, write down your top 3 hypotheses. What would you check first?