Investigation & Containment

Investigation

Treat the states and the cloud as evidence, not memory.

Cloud track - you read this, you do not run it. The commands below were run by the engineers in the story, Ana and Boris, on their own laptops against their team’s Hetzner project - not by you. You read what they ran and what came back (the output is shown right after each command), so you need no Hetzner account and nothing here touches your machine. You run the same investigation yourself, on your kind cluster, in lab step B4: there the two “laptops” are infra/terraform/state-lab and state-lab-b, and the cloud is the lab namespace.

Compare the memories. On each laptop, list what that state knows:
```
terraform state list
```
Both lists hold everything built before Monday. Ana’s also has hcloud_server.migration_db; Boris’s does not. Both have hcloud_server.worker - and terraform state show hcloud_server.worker reveals that they are two different servers: sre-workers-k2m on Ana’s laptop, sre-workers-p8x on Boris’s. Same code, two memories.

Compare the memories with reality. Ask the provider what exists, not Terraform:

hcloud server list -o columns=id,name

(That is the engineers’ command, on Hetzner. Your version, on kind, is in lab step B4 - there the provider is the Kubernetes API: kubectl --context kind-sre-control-plane -n lab get configmaps.)

The two lists do not speak the same language: the state lists addresses (hcloud_server.worker), the cloud lists IDs and names. Turn each state into IDs - on each laptop, from its own state:

terraform show -json | jq -r '.. | objects | select(.type? == "hcloud_server") | "\(.values.id)  \(.values.name)  \(.address)"'

.. walks the whole state, including servers created inside modules (kube-hetzner’s nodes are). Put the three answers side by side - one row per server that really exists in Hetzner, with the servers built before Monday left out (example IDs):

In Hetzner (`hcloud server list`)	In Ana’s state	In Boris’s state
51820001 `migration-db`	`hcloud_server.migration_db`	- not there
51820002 `sre-workers-k2m`	`hcloud_server.worker`	- not there
51820417 `sre-workers-p8x`	- not there	`hcloud_server.worker`

What to see:

The same address, hcloud_server.worker, is a different server in each state - sre-workers-k2m for Ana, sre-workers-p8x for Boris.
sre-workers-p8x exists in Hetzner but not in Ana’s state - the state the team will keep. That is the server Terraform has forgotten.

Why Ana’s state: it is the more complete one - it knows both of Monday’s servers. Every server in Hetzner that it does not contain is an object Terraform has forgotten. Here that is 51820417, the worker Boris’s apply created. It keeps running and costing money, and Terraform cannot connect it to anything: its address already belongs to sre-workers-k2m, so no plan will ever mention the extra worker. A forgotten object whose address no state tracks is worse the other way round - the next plan proposes to create it again, which is exactly what Boris’s plan did with migration-db.

The platform uses the same principle - ask the provider, not Terraform - in a simpler form after every Hetzner destroy: scripts/hcloud-leftovers.sh compares no IDs, it lists eight kinds of object in the project - servers, volumes, load balancers, primary IPs, floating IPs, networks, firewalls and placement groups - and fails if any is left. Servers, volumes, load balancers, floating IPs and primary IPv4 addresses cost money; a primary IPv6 address is not charged by itself; networks, firewalls and placement groups are free, but they block the next clean create. It can, because after a destroy the project should hold nothing at all - including the volumes and load balancers the cluster created itself, which were never in any state.

Find out how the gap appeared. The state is a file on a laptop, so every laptop that ever ran an apply has its own version. Nothing tells Boris’s Terraform that Ana’s is newer - there is no shared place to look, and nothing that stops two applies from running at the same time.

The root cause is where the state lives, not the plan or the review. A plan is only as good as the memory it is compared with.

Containment

Stop all applies for this infrastructure until one memory is agreed.
Pick the most complete state - here Ana’s, which knows both of Monday’s servers - and move it to a shared backend with locking (next lesson). From then on, every laptop and every CI job reads and writes that one state.
Deal with what no state knows. For each such object, decide deliberately: bring it under Terraform with terraform import if it should exist, or delete it after you have checked that it holds nothing you need.
Delete the local copies of the state. They are out of date and they hold secrets in plain text.
Prove it: from each laptop, terraform plan returns no changes - everyone now has the same memory, and it matches reality.

Pause and Predict: What would have prevented this incident entirely?