Incident Hook

A teammate commits a plaintext API key to fix a failing deploy quickly. The key is exposed in Git history, CI logs, and local clones. A simple git revert is not enough because the secret is already leaked and remains in the repository’s history.

Result: Response now includes rotation, audit, and cross-team coordination under pressure.

Observed Symptoms

What the team sees first:

• The deploy is unblocked, but the secret is visible in the PR diff.
• The same value may now exist in PR tooling, CI logs, and local clones.

Concrete evidence of the leak:

# diff view of the mistake
apiVersion: v1
kind: Secret
metadata:
  name: backend-api-key
stringData:
  # ❌ CRITICAL: Plaintext key committed directly to Git
  API_KEY: "sk_live_51P2x..." 

The visible file change is only the smallest part of the incident. Responders cannot tell immediately how many copies of the credential exist across the team’s workstations.

Confusion Phase

The first instinct is usually to revert the commit and move on. That is where teams lose time. The real questions are:

• Is the credential still valid?
• Where else was it copied while exposed?
• Has the replacement path already created a second unsafe secret?

Leak Response Mini-Runbook (First 30 Minutes)

1. Rotate/Revoke: Invalidate the exposed credential immediately at the source.
2. Invalidate Sessions: Invalidate active sessions or tokens that depend on the secret.
3. Identify Leak Surface: Map where the secret might have been copied (Git history, CI logs, chat, local copies).
4. Notify Owners: Open an incident record and notify affected service and security owners.
5. Secure Replacement: Confirm the replacement secret is distributed via the encrypted path (SOPS) only.

Pause and Predict: Before reading the investigation, write down your top 3 hypotheses. What would you check first?