The Incident: Two Laptops, One Cloud

Incident Hook

A team runs its Hetzner infrastructure with Terraform. The code lives in Git, and every change goes through a pull request. The state - Terraform’s record of what it built - lives where Terraform puts it by default: in a file, terraform.tfstate, on the laptop of whoever ran the apply. It is git-ignored, as it should be, because it holds secrets.

Monday. Ana adds two things for a database migration: a server called migration-db, and one more worker node. Worker names get a generated suffix - the same way this course’s own Hetzner nodes are called sre-workers-ezz - so the new worker becomes sre-workers-k2m. The pull request is reviewed and merged, and Ana applies it from their laptop:

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Terraform records both new servers in Ana’s terraform.tfstate - on Ana’s laptop.

Tuesday. Ana is on leave. Boris has to open a firewall port. Boris pulls main, makes the change and runs a plan on their own laptop. Boris’s Terraform has its own state: an older copy that has never heard of Monday’s servers. So the plan says:

  # hcloud_server.migration_db will be created
  # hcloud_server.worker will be created
  # hcloud_firewall.cluster will be updated in-place
Plan: 2 to add, 1 to change, 0 to destroy.

Boris and the reviewer read it. Both servers are in the merged code, so “create them” looks as if someone simply forgot to apply on Monday. They approve. The apply stops with an error - a server called migration-db already exists - but not before it has changed the firewall and created the worker. The worker’s name got a new suffix, sre-workers-p8x, so nothing collided: there are now two new workers, and no error mentions the second one.

Three weeks later. The migration is done, and Ana scales the workers back down with Terraform. Ana’s state knows one new worker, so one worker is removed. Boris’s worker keeps running - and keeps being billed - until someone reads the invoice.

The same story, one row per apply. Read each row left to right: what the engineer wanted, what their plan said, what Hetzner actually did, and what their own state file remembers afterwards.

Diagram: Monday - Ana; Wants migration-db + one worker; Plan, with Ana's state 2 to add; Hetzner creates migration-db sre-workers-k2m; Ana's state remembers migration-db, k2m; Tuesday - Boris (Ana is on leave); Wants one firewall port, nothing else; Plan, with Boris's week-old state 2 to add, 1 to change; Hetzner: firewall changed sre-workers-p8x created migration-db: already exists ✗; Boris's state remembers firewall, p8x; Three weeks later - Ana; Wants the extra worker gone; Plan, with Ana's state 1 to destroy: k2m; Hetzner removes sre-workers-k2m; sre-workers-p8x still runs and is billed - Ana's state does not know it

The red boxes are the incident: a plan that wanted far more than Boris asked for, and a server that Ana’s plan leaves running, because her state does not remember it - only Boris’s does.

Observed Symptoms

What the team sees:

  • Boris’s apply ends with an error about a name that is already in use - and with a partial result: the firewall change and one server went through.
  • terraform state list on Ana’s laptop and on Boris’s laptop shows different objects for the same code.
  • The Hetzner console shows one more worker than the code describes.
  • Ana’s scale-down finished cleanly, and a server nobody asked for is still running.

Confusion Phase

Nobody did anything wrong by their own lights:

  • The plan was read and approved. It showed exactly what Boris’s Terraform was going to do.
  • The code in Git was correct.
  • Each apply did what its plan said.

What went wrong is that the code was shared and the memory was not. Terraform decides what to create by comparing the code with its state. Two laptops meant two states - two different memories of the same infrastructure - and Boris’s memory was a week old.

The team now has to answer:

  • Which state is the truth - Ana’s, Boris’s, or neither?
  • Which objects in the cloud does no state know about at all?
  • How do two people - or two CI jobs - stop working from different memories, or writing the same memory at the same time?

What AI Would Propose (Brave Junior)

  • “The server already exists - add -target and apply the rest.”
  • “Send me your terraform.tfstate, I’ll copy it over mine.”
  • “Delete the extra worker in the console.”

Why This Is Dangerous

  • -target makes the plan smaller, not the memory right. The next full plan runs into the same gap.
  • Copying state files by chat or email spreads secrets - the state holds them in plain text - and the copy is out of date the moment either person applies again.
  • Deleting in the console without checking can remove the wrong server, and leaves both states wrong.
  • None of these stop it from happening again next week.

Pause and Predict: Boris’s plan was read and approved. Before reading the investigation, write down which lines in it should have stopped the review - and what you would have checked before saying yes.