Workflow: Shared State and Reviewed Plans

Terraform’s Memory: the State

Terraform does not look at your cloud and guess what belongs to you. It keeps a state: a file that maps every resource in the code (hcloud_server.worker) to the real object it created (server sre-workers-k2m, ID 1234567), with the values it last saw. Everything Terraform decides starts from that memory:

• a resource in the code but not in the state → create it;
• in both → compare, and update or replace it if they differ;
• in the state but no longer in the code → destroy it.

So Terraform cannot manage what its state does not list. In the incident, Boris’s state did not list Monday’s servers, so Boris’s Terraform wanted to create them again - and could not remove the second worker later, because Ana’s state never knew it. The state is not a cache you can rebuild at will. It is the only thing that connects the code to the real objects.

How a Plan Is Made

terraform plan compares three things:

1. The code - what you want (*.tf).
2. The state - what Terraform built last time.
3. The real objects - Terraform first refreshes: it asks the cluster or the cloud API what each object in the state looks like right now.

The plan is the difference between the code and the refreshed reality. Note what the refresh can see: only objects that are already in the state. It notices a firewall rule changed by hand on a firewall it knows. It cannot notice a server it has never heard of - that is why Boris’s plan was so confident.

terraform apply tfplan works differently. It does not refresh again - it executes the decisions saved in the file. The only check it makes is that the state is still the one the plan was made from (the same lineage and serial number); if another apply has written the state since, it stops with Saved plan is stale.

A change made by hand after the plan never writes the state, so the serial stays the same and the saved decisions run over it.

Both halves lead to the same two rules. There must be one memory that every plan compares with. And what you apply must be exactly the plan someone read, made recently enough that reality has not moved under it.

Level 1: One Shared, Locked State

Keep the state in a shared backend (backend). A backend is where Terraform keeps its state. The default, local, is a file next to the code - fine for one person on one machine, and the cause of the incident as soon as there are two. A shared backend - an S3 bucket, Terraform Cloud, a database - gives every laptop and every CI job the same memory. The Hetzner module keeps its state in a Cloudflare R2 bucket; the kind module keeps a local file, because a local lab cluster has exactly one owner.

How a state becomes shared. Three things change - none of them is the infrastructure itself.

First, a backend block in the code. Without one, Terraform uses local. The Hetzner module’s block points to an S3-compatible bucket in Cloudflare R2:

    # Cloudflare R2 S3-compatible endpoint (account-specific).
    endpoints = {
      s3 = "https://99c9887cccb1cb265d748f267999af47.r2.cloudflarestorage.com"
    }

    # Required for non-AWS S3 backends: R2 has no AWS account, region or STS to check.
    use_path_style              = true
    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true

    # Use S3-native locking (no DynamoDB): a lock object next to the state, written with a
    # conditional write - a second plan or apply waits or fails instead of overwriting.
    use_lockfile = true
  }
}

bucket and key say where the state lives: the object tfstate/hcloud/cluster.tfstate in the bucket sre. endpoints points the S3 protocol at R2 instead of AWS (the account ID in it is not a secret), and use_lockfile puts the lock next to the state (below). The block is committed to Git, so everyone who clones the repository points at the same state.

Second, one move. terraform init -migrate-state reads the local state and uploads it to the backend. From then on the local file is not used.

Third, credentials outside the code. Access to the bucket comes from environment variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the S3 protocol’s names for an R2 key; nothing secret is in the block. In CI the key is a GitHub Actions secret. A person who runs Terraform from a workstation gets their own key from the bucket’s owner - an R2 API token scoped to that one bucket (Cloudflare dashboard: R2, Manage API tokens) - keeps it in a git-ignored env file (the platform loads .env.local with source ./load-env.sh), and only then runs terraform init. Every key is personal, so one can be revoked without breaking the others.

A second developer clones the repository, sets their own key and runs terraform init. Terraform connects to the same object and copies nothing: every plan and apply, from either laptop, reads and writes that one state. The lab does the same on a small scale in a separate exercise, infra/terraform/state-lab: its backend is an S3-compatible bucket in the MinIO that runs in your kind cluster. The kind module itself keeps its local state.

Lock it (backend). One shared memory creates a new risk: two writers at once. Each run reads the state, changes objects and writes its own version back - so if two applies overlap, the second write silently loses the first one’s objects. Terraform therefore takes a lock before it plans or applies, and releases it at the end:

• for a local state, a file next to it, .terraform.tfstate.lock.info;
• for the Hetzner module, an object tfstate/hcloud/cluster.tfstate.tflock next to the state in R2 (use_lockfile = true). Terraform creates it with a conditional write - “only if it does not exist yet” - so of two runs that start together, exactly one succeeds.

A second run finds the lock and stops with Error acquiring the state lock, telling you who holds it. The lock is held for the whole run - also while an apply waits for you to type yes. The principle is one writer at a time.

Keep the backend outside what it manages. If the state lived inside the cluster Terraform builds, destroying the cluster would destroy Terraform’s memory of it. The lab uses the MinIO inside your kind cluster as a backend because it is already there - a shortcut for practice, not a pattern.

Protect it like a secret. The state holds passwords and keys in plain text (see Sensitive Is Not Secret below). A shared backend is also one place to control who may read it: only the people and the CI job that hold a key for the bucket can, and no key is ever given to the cluster.

Level 2: Apply Only What You Have Seen

A shared, locked state makes every plan compare with the same, current memory. It does not yet guarantee that what runs is what a person read and approved. That is the second level.

A saved plan is a snapshot. A plan saved with -out is exact - the reviewer approves precisely what will run - and it gets older every minute. Imagine a plan to remove an old firewall rule, reviewed in the morning. At 14:10 on-call adds a rule by hand for a partner who changed IP ranges. At 16:45 the morning plan is applied: Terraform sets the firewall’s rules to the list it computed in the morning - without the partner’s rule - and the saved-plan check does not object, because a change made in a console never writes the state. An age limit narrows this window; it does not close it - a hand change made ten minutes after the plan is just as invisible to the saved file. A fresh plan, made and read right before the apply, shows the hand changes made before it to objects already in its state (the refresh sees nothing else) and leaves only the short time between plan and apply - because apply tfplan runs the saved decisions, it never makes a new plan. The rest is an incident fix that goes into the code, not only into the console.

Where the plan lives. The state is shared through the backend; a saved plan is not. It is a file that belongs to one run, holds every secret the state holds, and is valid only for that state and that moment. So the question is not “where do we share the file?” but “who applies it?”:

• You apply, alone (your kind cluster): the plan stays on your disk, next to the code, and the guard below applies it and deletes it.
• A team applies (the Hetzner platform): the plan stays in CI. The plan job of an apply run - started by hand after the merge - makes it and keeps two things. The plan file (tfplan) - the one that holds the values and is applied - is encrypted (tfplan.enc), kept for one day, and only the apply job of the same workflow decrypts it, after checking its hash and age (In CI below). The readable plan - the run page, with the file’s SHA-256, and a plan.txt copy of it - is what the reviewer reads, and in a public repository anyone can read it too. Terraform prints sensitive values there as (sensitive value), so every secret has to be marked sensitive (or kept out with ephemeral) before it ever reaches a plan; anything unmarked is shown as it is.

Never share the plan file itself - not in Git, a chat, a ticket or a shared drive, and never as an unencrypted build artifact: that is how this platform leaked three keys (Sensitive Is Not Secret). Services such as HCP Terraform or Atlantis follow the same idea: the run and its plan stay in one place, and people read and approve it there.

On your workstation: guard-terraform-plan.sh (workstation). The guard makes “plan, read, apply that plan, soon” mechanical:

• plan runs terraform plan -out and writes tfplan.meta with the time the plan was made.
• apply refuses to run without the plan file and its metadata, or when the plan is older than --max-age-minutes (default 120; the lab uses 60). It then applies exactly that plan file, and only after a successful apply deletes it with its metadata (a refused or failed apply keeps both): the plan is spent, and it holds sensitive values in plain text (see Sensitive Is Not Secret below). The metadata is read as data, never run as a script.

What the guard does not do: it does not know whether anyone read the plan, it does not see a hand change made inside the age limit, and one sed on tfplan.meta fools it. On a workstation it is a seatbelt, not a lock.

#!/usr/bin/env bash
Guardrail wrapper for Terraform plan/apply.
- `plan` creates a planfile and metadata marker.
- `apply` refuses to run unless a fresh planfile + metadata marker exist.

Examples:
  scripts/guard-terraform-plan.sh plan --dir infra/terraform/hcloud_cluster --out tfplan
  scripts/guard-terraform-plan.sh apply --dir infra/terraform/hcloud_cluster --out tfplan --max-age-minutes 60
EOF
}

if [[ $# -lt 1 ]]; then
  usage >&2
  exit 2
fi

if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
  usage >&2
  exit 0
fi

MODE="$1"
shift

WORKDIR=""
PLAN_FILE="tfplan"
MAX_AGE_MINUTES="120"

if [[ -z "${WORKDIR}" ]]; then
  echo "[guard-tf] --dir is required" >&2
  usage >&2
  exit 2
fi

if ! command -v terraform >/dev/null 2>&1; then
  echo "[guard-tf] terraform not found in PATH" >&2
  exit 1
fi

if ! [[ -d "${WORKDIR}" ]]; then
  echo "[guard-tf] directory not found: ${WORKDIR}" >&2
  exit 1
fi

PLAN_PATH="${WORKDIR}/${PLAN_FILE}"
META_PATH="${PLAN_PATH}.meta"

case "${MODE}" in
  plan)
    terraform -chdir="${WORKDIR}" init -input=false
    terraform -chdir="${WORKDIR}" plan -input=false -lock-timeout=5m -out "${PLAN_FILE}"
    # The metadata the apply step reads, as key=value data: when the plan was made, for which
    # directory and plan file.
    {
      echo "created_at_epoch=$(date +%s)"
      echo "workdir=${WORKDIR}"
      echo "plan_file=${PLAN_FILE}"
    } > "${META_PATH}"
    echo "[guard-tf] plan created: ${PLAN_PATH}"
    echo "[guard-tf] metadata created: ${META_PATH}"
    ;;
  apply)
    if [[ ! -f "${PLAN_PATH}" ]]; then
      echo "[guard-tf] missing plan file: ${PLAN_PATH}" >&2
      echo "[guard-tf] run: scripts/guard-terraform-plan.sh plan --dir ${WORKDIR} --out ${PLAN_FILE}" >&2
      exit 1
    fi
    if [[ ! -f "${META_PATH}" ]]; then
      echo "[guard-tf] missing plan metadata: ${META_PATH}" >&2
      echo "[guard-tf] refusing apply without plan marker" >&2
      exit 1
    fi

    # Read the metadata as data, never run it: sourcing would execute whatever
    # the file contains. A missing, non-decimal or future time is a refusal:
    # a time after now (or one too large to fit) would give a negative age
    # that passes the age check.
    CREATED_AT_EPOCH="$(sed -n 's/^created_at_epoch=//p' "${META_PATH}" | head -n 1)"
    NOW_EPOCH="$(date +%s)"
    if ! [[ "${CREATED_AT_EPOCH}" =~ ^[1-9][0-9]{0,11}$ ]] || (( CREATED_AT_EPOCH > NOW_EPOCH )); then
      echo "[guard-tf] invalid created_at_epoch in ${META_PATH}" >&2
      echo "[guard-tf] re-run plan before apply" >&2
      exit 1
    fi
    AGE_SECONDS="$((NOW_EPOCH - CREATED_AT_EPOCH))"
    AGE_MINUTES="$((AGE_SECONDS / 60))"

    if (( AGE_MINUTES > MAX_AGE_MINUTES )); then
      echo "[guard-tf] plan is too old (${AGE_MINUTES}m > ${MAX_AGE_MINUTES}m)" >&2
      echo "[guard-tf] re-run plan before apply" >&2
      exit 1
    fi

    terraform -chdir="${WORKDIR}" apply -input=false "${PLAN_FILE}"
    echo "[guard-tf] apply completed using ${PLAN_PATH}"
    # The applied plan is spent (the state serial moved on) and holds sensitive
    # values in plain text: do not leave it on disk.
    rm -f "${PLAN_PATH}" "${META_PATH}"
    echo "[guard-tf] removed ${PLAN_PATH} and its metadata"
    ;;
  *)
    echo "[guard-tf] unknown mode: ${MODE}" >&2
    usage >&2
    exit 2
    ;;
esac

In CI: the Hetzner workflow (CI). .github/workflows/terraform-hcloud.yml is where the rule becomes enforceable:

1. A merge to main only plans: the run shows whether the change plans cleanly, and ends. Applying is a separate, deliberate run - Run workflow on main with apply ticked. That run’s plan job makes a fresh plan and writes the plan’s Plan: line, the full plan, the SHA-256 of the plan file and the time into the run’s summary - the same page where the approver clicks Review deployments. The plan file itself leaves the job only encrypted.
2. The apply job waits for a person’s approval in the production environment. The wait for a person comes from the environment’s required reviewers in GitHub, not from environment: production in the workflow file: without them the job would not wait for anyone - other environment rules, such as which branches may deploy, could still stop or delay it, but none of them asks a person. The platform repository has required reviewers set.
3. Before it applies, the apply job checks that the decrypted file has the same SHA-256 as the one shown in the summary, and that the plan is not older than 60 minutes. If either check fails, it applies nothing and tells you to run the workflow again for a fresh plan.

Two details matter. The plan in the pull request is not the one that runs: the apply run makes a fresh plan, and that is what you approve. And the hash turns “I read a plan” into “I read this plan” - the approver can see that the file about to run is byte for byte the one described on the page.

Drift: When Reality Moves Without Terraform

Drift is any difference between the code, the state and the real system - usually a change made by hand. terraform plan -detailed-exitcode answers in one number - a number, not text, so a script or a CI job can act on it without parsing the plan:

A hand change is allowed in an incident; leaving it only in the console is not. Every drift gets one of three answers, the same day:

• Revert - the hand change was a mistake: apply the code, and the drift is gone.
• Codify - the hand change is right: put it into the code, and a fresh plan shows no change for it.
• Stop - the plan wants to destroy or replace something that holds data: do not apply, and treat it as an incident.

Check the Code Before It Leaves Your Machine

pre-commit is a small framework that Git calls before every commit (make install-hooks registers it once). It reads .pre-commit-config.yaml and runs each hook whose file pattern matches the files you are committing; if any hook fails, the commit is not made. For .tf files three hooks run:

• terraform fmt -check - the code is in the one standard layout, so a review diff shows real changes, not whitespace.
• terraform validate - the code is internally consistent: every reference, type and required argument checks out. The hook initialises each module in a throw-away directory without a backend, so it never reads a state and needs no credentials.
• Checkov - a static security scan: it reads the code, not the cluster, and reports settings known to be risky, such as a secret in plain text or a network rule open to the world. Checkov is a required tool (make check-tools lists it); without it the hook fails instead of skipping, so the scan cannot quietly disappear.

The principle is fail fast and cheap: a mistake caught on your machine costs seconds, the same mistake caught in CI costs a push and a wait, and in production it costs an incident. But hooks run on your machine, and git commit --no-verify skips them. That is why CI checks again: today the Hetzner workflow repeats fmt and validate; moving the rest into CI is part of Chapter 05.

default_install_hook_types:
  - pre-commit
  - pre-push
  - pre-merge-commit
  - prepare-commit-msg

repos:
  - repo: local
    hooks:
      - id: master-branch-check
        name: Protected branch guard
        entry: scripts/pre-commit-master-check.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [pre-commit, pre-push, pre-merge-commit]
        args:
          - --protected=master
          - --protected=main

      - id: prevent-amend-after-push
        name: Prevent amending pushed commits
        entry: scripts/prevent-amend-after-push.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [prepare-commit-msg]

  - repo: local
    hooks:
      - id: guard-kube-context-test
        name: guard-kube-context tests (fake kubectl, no cluster)
        entry: tests/guard-kube-context.test.sh
        language: script
        files: ^(scripts/guard-kube-context\.sh|tests/guard-kube-context\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: merge-kubeconfig-test
        name: merge-kubeconfig tests (throw-away HOME, no cluster)
        entry: tests/merge-kubeconfig.test.sh
        language: script
        files: ^(infra/terraform/kind_cluster/scripts/merge-kubeconfig\.sh|tests/merge-kubeconfig\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kube-context-hook-test
        name: agent kube-context hook tests (text only, no cluster)
        entry: tests/kube-context-hook.test.sh
        language: script
        files: ^(scripts/agent-hooks/kube-context-hook\.sh|tests/kube-context-hook\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: prevent-amend-after-push-test
        name: amend guard tests (throw-away repository, no GitHub)
        entry: tests/prevent-amend-after-push.test.sh
        language: script
        files: ^(scripts/prevent-amend-after-push\.sh|tests/prevent-amend-after-push\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-sops-encrypted-test
        name: sops-encrypted check tests (throw-away repository, no sops)
        entry: tests/check-sops-encrypted.test.sh
        language: script
        files: ^(scripts/check-sops-encrypted\.sh|tests/check-sops-encrypted\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-app-security-test
        name: app-security check tests (throw-away copy of flux/, no cluster)
        entry: tests/check-app-security.test.sh
        language: script
        files: ^(scripts/check-app-security\.sh|tests/check-app-security\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-image-tags-test
        name: image-tag check tests (throw-away copy of flux/apps)
        entry: tests/check-image-tags.test.sh
        language: script
        files: ^(scripts/check-image-tags\.sh|tests/check-image-tags\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: postgres-restore-manifest-test
        name: restore manifest tests (stub kubectl, no cluster)
        entry: tests/postgres-restore-manifest.test.sh
        language: script
        files: ^(scripts/postgres-restore-manifest\.sh|tests/postgres-restore-manifest\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-policy-exceptions-test
        name: policy-exception check tests (throw-away copy, no cluster)
        entry: tests/check-policy-exceptions.test.sh
        language: script
        files: ^(scripts/check-policy-exceptions\.sh|tests/check-policy-exceptions\.test\.sh|tests/kyverno/exceptions\.yaml)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-checkov-skips-test
        name: Checkov skip check tests (throw-away copy, no checkov)
        entry: tests/check-checkov-skips.test.sh
        language: script
        files: ^(scripts/check-checkov-skips\.sh|tests/check-checkov-skips\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: capstone-fault-test
        name: capstone fault tests (throw-away copy, no cluster, no GitHub)
        entry: tests/capstone-fault.test.sh
        language: script
        files: ^(scripts/capstone-fault\.sh|tests/capstone-fault\.test\.sh|scripts/check-app-(security|resources)\.sh|flux/apps/backend/(base/deployment|develop/kustomization)\.yaml|flux/infrastructure/network-policies/base/allow-(backend-egress|postgres-ingress)-(postgres|backend)\.yaml)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-credentials-test
        name: credential registry check tests (throw-away copy, no secrets)
        entry: tests/check-credentials.test.sh
        language: script
        files: ^(scripts/check-credentials\.sh|tests/check-credentials\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kyverno-policies-test
        name: Admission guardrails give the expected verdicts (kyverno test, no cluster)
        entry: tests/kyverno-policies.test.sh
        language: script
        files: ^(flux/infrastructure/policy/packs/admission-guardrails/.*|tests/kyverno/.*|tests/kyverno-policies\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kyverno-supply-chain-test
        name: Supply-chain policies give the expected verdicts on real images (kyverno test, network)
        entry: tests/kyverno-supply-chain.test.sh
        language: script
        files: ^(flux/infrastructure/policy/packs/supply-chain/.*|tests/kyverno-supply-chain/.*|tests/kyverno-supply-chain\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: k3s-audit-policy-test
        name: The k3s audit policy loads in the pinned k3s and logs what it promises (Docker, network)
        entry: tests/k3s-audit-policy.test.sh
        language: script
        files: ^(infra/terraform/hcloud_cluster/audit-policy\.yaml|infra/terraform/hcloud_cluster/terraform\.tfvars|tests/k3s-audit-policy\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: tool-versions-test
        name: The tests run the Kyverno and Prometheus versions the cluster runs (helm show)
        entry: tests/tool-versions.test.sh
        language: script
        files: ^(flux/infrastructure/policy/kyverno/release\.yaml|flux/infrastructure/observability/kube-prometheus-stack/base/release\.yaml|\.github/workflows/pre-commit\.yml|tests/tool-versions\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: platform-alerts-test
        name: Platform alerts fire where they should, and only there; every PrometheusRule parses (promtool, no cluster)
        entry: tests/platform-alerts.test.sh
        language: script
        # Any manifest under flux/ may be a PrometheusRule (the test finds them by kind; ~1 s).
        files: ^(flux/.*\.ya?ml|tests/platform-alerts/.*|tests/platform-alerts\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: flux-kustomize-validate
        name: Flux kustomize validate
        entry: scripts/flux-kustomize-validate.sh
        language: script
        files: ^flux/.*\.ya?ml$
        pass_filenames: true
        require_serial: true
        stages: [pre-commit]

      - id: app-security
        name: App manifests keep the security rules (no SA token, no pprof, no chaos in production); no service links in any flux/ workload
        entry: scripts/check-app-security.sh
        language: script
        files: ^(flux/.*\.ya?ml|scripts/check-app-security\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: oidc-rbac
        name: OIDC RBAC binds only Dex org:team groups; write roles only in develop
        entry: scripts/check-oidc-rbac.sh
        language: script
        files: ^flux/infrastructure/security/(rbac|dex)/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: policy-exceptions
        name: PolicyExceptions are narrow, owned and not expired
        entry: scripts/check-policy-exceptions.sh
        language: script
        files: ^flux/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: credentials
        name: Every credential is in the registry, described and not due for rotation
        entry: scripts/check-credentials.sh
        language: script
        files: ^(docs/credential-(registry\.yaml|rotation\.md)|flux/secrets/.*|scripts/check-credentials\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: image-tags
        name: Each overlay's image tag matches its own ImagePolicy, with the setter comment
        entry: scripts/check-image-tags.sh
        language: script
        files: ^flux/apps/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: app-resources
        name: App resources (staging = production >= develop) fit the namespace quotas
        entry: scripts/check-app-resources.sh
        language: script
        files: ^flux/(apps|infrastructure/resource-management|infrastructure/data/cnpg-clusters)/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-fmt
        name: Terraform format check
        entry: terraform fmt -recursive -diff -check
        language: system
        files: \.tf$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-validate
        name: Terraform validate
        entry: scripts/terraform-validate.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

      - id: checkov-skips
        name: Every Checkov skip has a reason and is in infra/terraform/checkov-skips.txt
        entry: scripts/check-checkov-skips.sh
        language: script
        files: ^(infra/terraform/.*\.tf|infra/terraform/checkov-skips\.txt|scripts/check-checkov-skips\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-security
        name: Terraform security scan
        entry: scripts/terraform-security.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

  - repo: local
    hooks:
      - id: no-secrets
        name: Block sensitive files
        entry: scripts/block-secrets.sh
        language: script
        # The file NAME, not the whole path: scripts/merge-kubeconfig.sh is not a kubeconfig.
        # Terraform state and saved plans hold every secret in plain text (Chapter 02):
        # *.tfstate, its backups (*.tfstate.*), tfplan, tfplan.meta, *.tfplan.
        # The age private key decrypts every SOPS secret: *.agekey and its backups (*.agekey.*).
        files: (^|/)(kubeconfig(\.ya?ml)?|[^/]+\.(key|pem)|[^/]*\.agekey(\.[^/]+)?|credentials[^/]*|[^/]*\.env(\.[^/]+)?|[^/]*\.tfstate(\.[^/]+)?|tfplan(\.meta)?|[^/]*\.tfplan)$
        exclude: \.example$
        stages: [pre-commit]

      - id: sops-encrypted
        name: Secrets under flux/secrets/ are SOPS-encrypted
        entry: scripts/check-sops-encrypted.sh
        language: script
        files: ^flux/secrets/.*\.ya?ml$
        stages: [pre-commit]

  # Secrets by CONTENT in the staged changes (rules: .gitleaks.toml). The hooks above see file names
  # and SOPS structure only.
  # pre-commit builds the hook with Go on first use - it installs its own Go if none is present.
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1 # = GITLEAKS_VERSION in .github/workflows/secrets-guard.yml (the job checks it)
    hooks:
      - id: gitleaks
        stages: [pre-commit]

  - repo: https://github.com/koalaman/shellcheck-precommit
    rev: v0.11.0
    hooks:
      - id: shellcheck
        files: \.sh$
        args: [--severity=warning]
        stages: [pre-commit]

  - repo: https://github.com/adrienverge/yamllint
    rev: v1.38.0
    hooks:
      - id: yamllint
        files: \.ya?ml$
        args: [-d, relaxed]
        stages: [pre-commit]

Reading a Plan

The review is the part no tool does for you. Read every line that starts with a symbol:

• + create, ~ update in place, - destroy, -/+ destroy and create again (a replacement).
• A replacement of anything that holds data - a server, a volume, a database - stops the review until someone explains it.
• A create of something that should already exist - like Monday’s servers in Boris’s plan - means the state does not know it. Stop and find out which state is right before you apply.
• A change to a resource the pull request is not about means drift, or a plan made in the wrong place. Find out which before you apply.
• The last line, Plan: X to add, Y to change, Z to destroy, must match what the pull request says it does.

Sensitive Is Not Secret

sensitive = true on a variable or an output changes one thing: Terraform prints (sensitive value) instead of the value in its output. It does not encrypt anything. Unless the value is also ephemeral (below), it is still stored in plain text in two files:

• The saved plan. tfplan is a zip archive. It holds every input value that is not ephemeral, the planned values and a full copy of the state. terraform show tfplan hides the sensitive values; terraform show -json tfplan prints them.
• The state. Terraform keeps what it writes so it can compare it with the next plan. A generated password (random_password) is in the state, because Terraform must remember it to keep it stable.

This course’s own platform repository learned this in September 2026. The Hetzner workflow uploaded the saved plan as a build artifact, so that the apply job could replay exactly the reviewed plan. The repository is public, and so are its artifacts. The plan held the Hetzner API token, the nodes’ SSH private key and the SOPS age key - all three marked sensitive, none of them visible in any log. A review of the workflow found it. The artifact was deleted and all three keys were rotated. The values encrypted with the old SOPS key were replaced too, because the encrypted files are public and the old key would have opened them. One value was knowingly left as it was: the Uptrace DSN, a telemetry write key, accepted as a known risk until the observability chapter changes it.

Protect the values in this order:

1. Treat plan files and state as secrets. Both are git-ignored in the platform repository. A plan leaves the machine that made it only encrypted: the Hetzner workflow encrypts it with openssl and a passphrase kept in a GitHub secret, and the apply job decrypts it. The key to the state bucket is never given to the cluster.

   On kind the state is a local file, so the risk is smaller - but not zero if other people have accounts on your laptop. Terraform writes the state, its backup and saved plans readable by everyone (-rw-r--r-- with the usual umask 022), and on macOS every regular user is in the group that may enter your home directory. A chmod 600 on the state is not enough: the state keeps it, but the next apply writes the backup - a full copy of the previous state - as a new file with the default permissions, and every saved plan is new too. Close the directory instead - chmod 700 infra/terraform/kind_cluster protects everything in it, new files included.

2. Keep secrets out of the plan and the state. An ephemeral variable (Terraform 1.10 and later) is never written to either. Terraform allows it only where it does not store the value, such as a write-only attribute (Terraform 1.11 and later - the modules require it). Both cluster modules hand a SOPS key you supply (TF_VAR_sops_age_key) to the cluster this way:

   variable "sops_age_key" {
     type      = string
     sensitive = true
     ephemeral = true
   }
   
   resource "kubernetes_secret_v1" "sops_age" {
     # ...
     data_wo          = { "age.agekey" = var.sops_age_key }
     data_wo_revision = var.sops_age_key_revision
   }
   

   Terraform cannot see a change in a write-only value. After you rotate the key, raise sops_age_key_revision, or Terraform never sends the new one.

   On kind you usually do not supply one: the local profile generates a key file on the first apply and reads it with data.local_file - so that default key is in the local state. A throwaway development key is an accepted trade-off; a key that protects anything real goes in through the variable. For that file there is no revision to raise: the kind module derives it from the file itself, so a new key file reaches the cluster on the next apply (Chapter 04 rotates it).

3. Better still, keep secrets out of Terraform. The Hetzner module does not take the nodes’ SSH private key at all: Terraform signs in to the nodes through ssh-agent, and only the public key is a Terraform value.

What stays is what Terraform must remember or read - for example the generated passwords, and on kind the default generated SOPS key. That is why the state is always a secret, whatever the variables say.

One Object, One Owner

Every field needs one owner. Two tools that manage the same field do not hide drift - they create it on every run, each undoing the other. Terraform creates the develop, staging, production and observability namespaces, but Flux writes their labels and annotations. If Terraform also managed those fields, every plan would try to undo Flux, and every Flux reconcile would undo Terraform. So the module tells Terraform to ignore them:

lifecycle {
  ignore_changes = [metadata[0].labels, metadata[0].annotations]
}

The price: a label changed by hand on those namespaces never shows up as drift. That is why the lab uses objects Terraform owns alone.

The End of the Lifecycle: Destroy (Cloud track)

terraform destroy removes what the state lists - and only that. In a cloud cluster, some objects are created not by Terraform but by controllers inside the cluster, on the cluster’s behalf:

• the CSI driver creates a Hetzner volume for every PersistentVolumeClaim (the databases’ disks).

The ingress’s load balancer is not one of them on purpose: kube-hetzner creates it as a Terraform resource (sre-custom), and Traefik’s Service only configures it - so the destroy removes it like any other resource. Where a provider’s controller creates the load balancer from the Service instead, it joins the list above.

Terraform never sees them, so a plain destroy removes the servers and leaves those behind - billed until someone notices. An earlier destroy of this course’s cluster left six volumes like that. The Hetzner module therefore runs a pre-destroy step first, in a fixed order, while the cluster can still clean up after itself:

1. suspend Flux, so it does not recreate what is being removed;
2. stop Kyverno and remove the admission webhooks, so no webhook blocks a deletion;
3. delete the application namespaces and the remaining PersistentVolumeClaims - the CSI driver deletes the volumes;
4. remove the Flux instance and its namespace;
5. then let Terraform destroy the servers, the network and the rest.

Some objects do not go quietly: a namespace stays Terminating while an object in it still has a finalizer - a marker that says “a controller must clean me up first”. If that controller is already gone, nothing ever removes the marker. After three minutes the script clears the finalizers itself. On 2026-09-27 the pre-destroy took 12 minutes 37 seconds, and about six of those were two namespaces waiting on finalizers.

Finally, the platform does not trust its own destroy: scripts/hcloud-leftovers.sh asks the Hetzner API for anything left in the project and fails loudly if there is. That run answered Hetzner project is empty. The principle is the same as in the incident: ask the provider, not Terraform’s memory, what exists.

The Hetzner state also holds the cluster’s admin kubeconfig - one more reason the state is treated as a secret.

Safe Workflow (Step-by-Step)

1. One memory. Everyone and every CI job uses the same shared, locked backend. On your kind cluster the local state is fine - you are its only user.
2. Start clean. terraform plan -detailed-exitcode returns 0. If it returns 2 and you have not changed anything, deal with that drift first.
3. Change the code and plan it. scripts/guard-terraform-plan.sh plan --dir infra/terraform/kind_cluster --out tfplan, then terraform -chdir=infra/terraform/kind_cluster show tfplan and read it. Short form: make kind-plan.
4. Apply that plan, soon. scripts/guard-terraform-plan.sh apply --dir infra/terraform/kind_cluster --out tfplan --max-age-minutes 60. Short form: make kind-apply. If the guard refuses, plan again and read the new plan - do not raise the limit. In CI the same rule is the hash and age check before the apply.
5. Prove it. terraform plan -detailed-exitcode returns 0 again.

In these modules Terraform never uses your current kubectl context. Its Kubernetes and Helm providers take the address and client certificate straight from the cluster resource it created (kind_cluster.sre), and the steps that install Flux use the module’s own kubeconfig.yaml, so Terraform always talks to the cluster it built. The kubectl commands you run next to it are different: they would use the shared current context, so every one of them names the cluster - --context kind-sre-control-plane (Chapter 01).

Instructions for Your AI Agent

The rules this chapter adds to the list from Chapter 01. Here most of them are held by the person who reads the plan - which is the point of this chapter.

1. Never run terraform apply without a saved plan that a person has read. The agent plans (make kind-plan) and stops; the person applies (make kind-apply). Enforced by: the guard behind make kind-apply (scripts/guard-terraform-plan.sh) refuses a missing plan or one older than 60 minutes - it does not check that anyone read or approved it; that part is an instruction. A bare terraform apply is not blocked on your machine - instruction only. On Hetzner, CI applies only the plan a person approved, with the hash shown on the run page.
2. Never -auto-approve, and never terraform destroy on your own. Enforced by: nothing on your machine - instruction only.
3. Never commit, paste or upload a state or a plan file (*.tfstate*, tfplan, tfplan.meta, *.tfplan). Enforced by: .gitignore, plus the no-secrets pre-commit hook and the Secrets guard CI job, which refuse them even after git add -f. Pasting into a chat: instruction only.
4. A plan that destroys or replaces something that holds data, or creates something that should already exist: stop and report - do not apply. Enforced by: the person who reads the plan.
5. Drift is reported, not fixed. Check it before and after a change (make kind-drift). Exit 2 without a code change of yours goes to the person - it may be drift, code merged but not applied, or different TF_VAR_* - who decides: revert, codify or stop. Enforced by: instruction only.

The full list, chapter by chapter, is docs/agent-rules.md in your fork of the platform repository.

This builds on: Blast Radius & the Shape of Safety (Chapter 01) - rules I, II and IV: verify the target, plan before apply, evidence before merge and after deploy. This enables: GitOps (Chapter 03) - Terraform hands the cluster to Flux; from there on, Git drives everything it deploys.