The Incident: The Hooks Nobody Ran

Incident Hook

Tuesday, 14:10. Boris renames a variable in the kind module, infra/terraform/kind_cluster/, and misses one place that still uses the old name. The terraform-validate hook would catch it - but on his laptop it takes forty seconds, and it has been slow all week. He commits with git commit --no-verify, pushes his branch and opens a pull request.

The pull request shows one check: the Secrets guard, green. All checks have passed. Ana is in a meeting; the change is one line; Boris merges.

Wednesday, 09:00. Ana rebuilds her kind cluster, as Chapter 00 shows:

$ make kind-plan
Error: Reference to undeclared input variable

So does everyone who pulls main that day. The rebuild of every lab cluster is broken - by a change that one hook would have refused in forty seconds.

Result: the only place that checked this Terraform was Boris’s laptop - and one flag skipped it.

The gap was real: until October 2026, CI on this platform ran only the secret checks, the Flux checks and the plan of the Hetzner module. terraform validate of the kind module, shellcheck, yamllint, checkov and the tests of the scripts ran on laptops only. Boris’s variable is the part this story adds.

Observed Symptoms

What the pull request showed on Tuesday:

All checks have passed
1 successful check
  ✓  Secrets guard / secrets (pull_request)   Successful
This branch has no conflicts with the base branch
                                              [ Merge pull request ]

Green - and it said nothing about Terraform. A check that does not exist cannot fail.

Confusion Phase

The pull request looked like every other one:

• the check was green - the only one there was;
• the change was one line, and the diff looked right;
• the hooks exist, and everyone “usually” runs them.

So who was supposed to catch it - Boris, Ana, or CI?

What AI Would Propose (Brave Junior)

• “The hooks are slow - alias git commit to git commit --no-verify, CI checks everything anyway.”
• “Skip the pull request for one-line changes - push and fix it later if something breaks.”
• “Run terraform apply from your laptop, it is faster than waiting for CI.”

Why This Is Dangerous

• “CI checks everything anyway” was not true - and nobody knew, because a missing check is not red. Before you trust CI, know which checks it runs.
• A one-line change is still a change. Here one line broke every lab cluster; the size of the diff says nothing about the size of the impact.
• An apply from a laptop is seen by one person. Nobody else reads the plan, and it needs the tools and the credentials on that laptop. A plan on a pull request is read by everyone who reviews it - and needs neither.

Pause and Predict: Boris skipped the hooks on purpose. Which two things on GitHub would have stopped his pull request anyway - without depending on his laptop or on Ana’s attention?