Investigation & Containment

Investigation

Start from what is broken on main, then walk back to the pull request and ask what checked it. The commands use the GitHub CLI (gh) against the platform repository; -R names it, the way --context names a cluster.

1. What is broken, and since which commit? Run every check on main - the same command CI runs since the incident:

   git switch --detach origin/main
   SKIP=master-branch-check pre-commit run --all-files
   git log --first-parent --oneline -5 origin/main -- infra/terraform/kind_cluster
   

   In the incident, terraform-validate failed with Reference to undeclared input variable, and the log named the merge that brought it in. (SKIP leaves out the hook that refuses work on main: this is a look, not a commit.) On today’s main the same command passes; the lab reproduces the failure with a file of its own (B2).

2. Which checks did that pull request have?

   gh pr checks 82 -R safeops-course/sre
   

   Two lines: secrets pass, and the advisory CodeRabbit review. No Terraform check ran on it at all.

3. Which checks does CI run - and on which pull requests? Each workflow’s on: says when it runs:

   grep -A6 '^on:' .github/workflows/*.yml
   

   At the time, the Secrets guard ran on every pull request; Flux Diff only when flux/** changed, the Terraform plan only when the Hetzner module changed (paths: under pull_request:). A change to the kind module matched neither filter, so the only check that ran was the Secrets guard - which does not validate Terraform. No Terraform validation ran on Boris’s pull request. Today none of these checks has a path filter on pull requests. The command above shows today’s files; the platform’s workflows with the filters are in commit 6a4b5eb, the last one before they were removed:

   git show 6a4b5eb:.github/workflows/flux-diff.yml | grep -A9 '^on:'
   git show 6a4b5eb:.github/workflows/terraform-hcloud.yml | grep -A12 '^on:'
   

4. Which checks are required for a merge?

   gh api repos/safeops-course/sre/rules/branches/main \
     -q '.[] | select(.type=="required_status_checks") | .parameters.required_status_checks[].context'
   

   At the time: secrets. Every other check - had there been one - could have been red, and the button would still have been green. Today: hooks, secrets, validate and terraform-plan-gate.

The root cause is not Boris’s typo, and not his --no-verify alone. It is that the hooks were the only place those checks ran: skip them once, and nothing on GitHub looked again.

Containment

1. Fix main through a pull request - a revert of the merge, or the one-line fix. Its checks are what you trust now, so read them.
2. Tell everyone who pulled main: rebuild after the fix is merged.
3. Close the gap (the next page): one CI job runs every hook on every pull request, and that job is a required check. Skipping the hooks then only moves the failure from the laptop to the pull request - where everyone sees it.

A check that runs only on a laptop is a habit, not a guardrail. Every hook runs again in CI, and the checks that matter are required.

Pause and Predict: With every hook running in CI, what is the cheapest place left to catch Boris’s mistake - and why keep the hooks on the laptop at all?