Workflow: Every Check Twice

What CI Gives You

A pull request is where a change becomes visible to the team before it runs. CI is what fills it with facts:

The plan, without the tools or the keys. Planning the Hetzner platform needs Terraform, its providers, the Hetzner token, the keys to the state bucket and the SOPS key. CI has them as repository secrets; you have none of them. A team member pushes a branch to the repository, opens a pull request, and the plan appears on it - Plan: 1 to add, 0 to change, 1 to destroy and every line behind it. (A pull request from a fork gets no secrets: GitHub does not hand them to code from outside the repository, so its Terraform plan fails.)
The change and its effect, side by side. The diff shows what you edited; the plan shows what it would do to real servers. A rename that destroys and recreates a resource is obvious in the plan, invisible in the diff.
A discussion before anything is applied. The plan has run; reviewers comment on the exact line, ask why, suggest a fix. CodeRabbit adds a review of its own. No infrastructure has changed yet - changing your mind costs a commit.
The same checks for everyone, every time. Not “usually, on the laptops where the hooks are installed”.
A decision by the right people. Who may apply is set in GitHub: the environment production waits for one of its reviewers. Proposing a change needs none of those rights.
A record. Who changed what, why, who reviewed it and who approved the apply - the pull request and the run keep it.

The opposite is the shortcut from the incident’s AI proposals: run it from your laptop, see the result alone, and find out later what it broke.

Hooks on Your Machine

make install-hooks registers pre-commit with Git for four moments of a commit’s life. From then on Git calls it by itself:

Git hook	When Git runs it	What runs
`pre-commit`	before a commit is created	every check in `.pre-commit-config.yaml` for the staged files - secrets, Flux manifests, app rules, Terraform, shell, YAML - plus the branch guard
`prepare-commit-msg`	while the commit message is prepared - also for `git commit --amend`	`prevent-amend-after-push`: refuses to amend a commit that a remote branch already contains
`pre-push`	before `git push` sends anything	the branch guard: no push while you are on `main`
`pre-merge-commit`	before a local merge commit	the branch guard

The branch guard is the hook master-branch-check (script scripts/pre-commit-master-check.sh): no commit, push or merge commit on main or master. From this chapter on, your work goes on a branch.

Pre-commit configuration

Show .pre-commit-config.yaml

default_install_hook_types:
  - pre-commit
  - pre-push
  - pre-merge-commit
  - prepare-commit-msg

repos:
  - repo: local
    hooks:
      - id: master-branch-check
        name: Protected branch guard
        entry: scripts/pre-commit-master-check.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [pre-commit, pre-push, pre-merge-commit]
        args:
          - --protected=master
          - --protected=main

      - id: prevent-amend-after-push
        name: Prevent amending pushed commits
        entry: scripts/prevent-amend-after-push.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [prepare-commit-msg]

  - repo: local
    hooks:
      - id: guard-kube-context-test
        name: guard-kube-context tests (fake kubectl, no cluster)
        entry: tests/guard-kube-context.test.sh
        language: script
        files: ^(scripts/guard-kube-context\.sh|tests/guard-kube-context\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: merge-kubeconfig-test
        name: merge-kubeconfig tests (throw-away HOME, no cluster)
        entry: tests/merge-kubeconfig.test.sh
        language: script
        files: ^(infra/terraform/kind_cluster/scripts/merge-kubeconfig\.sh|tests/merge-kubeconfig\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kube-context-hook-test
        name: agent kube-context hook tests (text only, no cluster)
        entry: tests/kube-context-hook.test.sh
        language: script
        files: ^(scripts/agent-hooks/kube-context-hook\.sh|tests/kube-context-hook\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: prevent-amend-after-push-test
        name: amend guard tests (throw-away repository, no GitHub)
        entry: tests/prevent-amend-after-push.test.sh
        language: script
        files: ^(scripts/prevent-amend-after-push\.sh|tests/prevent-amend-after-push\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-sops-encrypted-test
        name: sops-encrypted check tests (throw-away repository, no sops)
        entry: tests/check-sops-encrypted.test.sh
        language: script
        files: ^(scripts/check-sops-encrypted\.sh|tests/check-sops-encrypted\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-app-security-test
        name: app-security check tests (throw-away copy of flux/, no cluster)
        entry: tests/check-app-security.test.sh
        language: script
        files: ^(scripts/check-app-security\.sh|tests/check-app-security\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-image-tags-test
        name: image-tag check tests (throw-away copy of flux/apps)
        entry: tests/check-image-tags.test.sh
        language: script
        files: ^(scripts/check-image-tags\.sh|tests/check-image-tags\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: postgres-restore-manifest-test
        name: restore manifest tests (stub kubectl, no cluster)
        entry: tests/postgres-restore-manifest.test.sh
        language: script
        files: ^(scripts/postgres-restore-manifest\.sh|tests/postgres-restore-manifest\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-policy-exceptions-test
        name: policy-exception check tests (throw-away copy, no cluster)
        entry: tests/check-policy-exceptions.test.sh
        language: script
        files: ^(scripts/check-policy-exceptions\.sh|tests/check-policy-exceptions\.test\.sh|tests/kyverno/exceptions\.yaml)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-checkov-skips-test
        name: Checkov skip check tests (throw-away copy, no checkov)
        entry: tests/check-checkov-skips.test.sh
        language: script
        files: ^(scripts/check-checkov-skips\.sh|tests/check-checkov-skips\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: capstone-fault-test
        name: capstone fault tests (throw-away copy, no cluster, no GitHub)
        entry: tests/capstone-fault.test.sh
        language: script
        files: ^(scripts/capstone-fault\.sh|tests/capstone-fault\.test\.sh|scripts/check-app-(security|resources)\.sh|flux/apps/backend/(base/deployment|develop/kustomization)\.yaml|flux/infrastructure/network-policies/base/allow-(backend-egress|postgres-ingress)-(postgres|backend)\.yaml)$
        pass_filenames: false
        stages: [pre-commit]

      - id: check-credentials-test
        name: credential registry check tests (throw-away copy, no secrets)
        entry: tests/check-credentials.test.sh
        language: script
        files: ^(scripts/check-credentials\.sh|tests/check-credentials\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kyverno-policies-test
        name: Admission guardrails give the expected verdicts (kyverno test, no cluster)
        entry: tests/kyverno-policies.test.sh
        language: script
        files: ^(flux/infrastructure/policy/packs/admission-guardrails/.*|tests/kyverno/.*|tests/kyverno-policies\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: kyverno-supply-chain-test
        name: Supply-chain policies give the expected verdicts on real images (kyverno test, network)
        entry: tests/kyverno-supply-chain.test.sh
        language: script
        files: ^(flux/infrastructure/policy/packs/supply-chain/.*|tests/kyverno-supply-chain/.*|tests/kyverno-supply-chain\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: k3s-audit-policy-test
        name: The k3s audit policy loads in the pinned k3s and logs what it promises (Docker, network)
        entry: tests/k3s-audit-policy.test.sh
        language: script
        files: ^(infra/terraform/hcloud_cluster/audit-policy\.yaml|infra/terraform/hcloud_cluster/terraform\.tfvars|tests/k3s-audit-policy\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: tool-versions-test
        name: The tests run the Kyverno and Prometheus versions the cluster runs (helm show)
        entry: tests/tool-versions.test.sh
        language: script
        files: ^(flux/infrastructure/policy/kyverno/release\.yaml|flux/infrastructure/observability/kube-prometheus-stack/base/release\.yaml|\.github/workflows/pre-commit\.yml|tests/tool-versions\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: platform-alerts-test
        name: Platform alerts fire where they should, and only there; every PrometheusRule parses (promtool, no cluster)
        entry: tests/platform-alerts.test.sh
        language: script
        # Any manifest under flux/ may be a PrometheusRule (the test finds them by kind; ~1 s).
        files: ^(flux/.*\.ya?ml|tests/platform-alerts/.*|tests/platform-alerts\.test\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: flux-kustomize-validate
        name: Flux kustomize validate
        entry: scripts/flux-kustomize-validate.sh
        language: script
        files: ^flux/.*\.ya?ml$
        pass_filenames: true
        require_serial: true
        stages: [pre-commit]

      - id: app-security
        name: App manifests keep the security rules (no SA token, no pprof, no chaos in production); no service links in any flux/ workload
        entry: scripts/check-app-security.sh
        language: script
        files: ^(flux/.*\.ya?ml|scripts/check-app-security\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: oidc-rbac
        name: OIDC RBAC binds only Dex org:team groups; write roles only in develop
        entry: scripts/check-oidc-rbac.sh
        language: script
        files: ^flux/infrastructure/security/(rbac|dex)/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: policy-exceptions
        name: PolicyExceptions are narrow, owned and not expired
        entry: scripts/check-policy-exceptions.sh
        language: script
        files: ^flux/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: credentials
        name: Every credential is in the registry, described and not due for rotation
        entry: scripts/check-credentials.sh
        language: script
        files: ^(docs/credential-(registry\.yaml|rotation\.md)|flux/secrets/.*|scripts/check-credentials\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: image-tags
        name: Each overlay's image tag matches its own ImagePolicy, with the setter comment
        entry: scripts/check-image-tags.sh
        language: script
        files: ^flux/apps/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: app-resources
        name: App resources (staging = production >= develop) fit the namespace quotas
        entry: scripts/check-app-resources.sh
        language: script
        files: ^flux/(apps|infrastructure/resource-management|infrastructure/data/cnpg-clusters)/.*\.ya?ml$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-fmt
        name: Terraform format check
        entry: terraform fmt -recursive -diff -check
        language: system
        files: \.tf$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-validate
        name: Terraform validate
        entry: scripts/terraform-validate.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

      - id: checkov-skips
        name: Every Checkov skip has a reason and is in infra/terraform/checkov-skips.txt
        entry: scripts/check-checkov-skips.sh
        language: script
        files: ^(infra/terraform/.*\.tf|infra/terraform/checkov-skips\.txt|scripts/check-checkov-skips\.sh)$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-security
        name: Terraform security scan
        entry: scripts/terraform-security.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

  - repo: local
    hooks:
      - id: no-secrets
        name: Block sensitive files
        entry: scripts/block-secrets.sh
        language: script
        # The file NAME, not the whole path: scripts/merge-kubeconfig.sh is not a kubeconfig.
        # Terraform state and saved plans hold every secret in plain text (Chapter 02):
        # *.tfstate, its backups (*.tfstate.*), tfplan, tfplan.meta, *.tfplan.
        # The age private key decrypts every SOPS secret: *.agekey and its backups (*.agekey.*).
        files: (^|/)(kubeconfig(\.ya?ml)?|[^/]+\.(key|pem)|[^/]*\.agekey(\.[^/]+)?|credentials[^/]*|[^/]*\.env(\.[^/]+)?|[^/]*\.tfstate(\.[^/]+)?|tfplan(\.meta)?|[^/]*\.tfplan)$
        exclude: \.example$
        stages: [pre-commit]

      - id: sops-encrypted
        name: Secrets under flux/secrets/ are SOPS-encrypted
        entry: scripts/check-sops-encrypted.sh
        language: script
        files: ^flux/secrets/.*\.ya?ml$
        stages: [pre-commit]

  # Secrets by CONTENT in the staged changes (rules: .gitleaks.toml). The hooks above see file names
  # and SOPS structure only.
  # pre-commit builds the hook with Go on first use - it installs its own Go if none is present.
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1 # = GITLEAKS_VERSION in .github/workflows/secrets-guard.yml (the job checks it)
    hooks:
      - id: gitleaks
        stages: [pre-commit]

  - repo: https://github.com/koalaman/shellcheck-precommit
    rev: v0.11.0
    hooks:
      - id: shellcheck
        files: \.sh$
        args: [--severity=warning]
        stages: [pre-commit]

  - repo: https://github.com/adrienverge/yamllint
    rev: v1.38.0
    hooks:
      - id: yamllint
        files: \.ya?ml$
        args: [-d, relaxed]
        stages: [pre-commit]

What --no-verify skips - and what it does not. git commit --no-verify skips the pre-commit hook: every check on the files. It does not skip prepare-commit-msg, so the amend guard still runs. git push --no-verify skips the pre-push branch guard. Either flag switches off part of the local protection: the checks of your files run again in CI on the pull request, and the rule against pushing to main is enforced by the repository’s ruleset - neither depends on your machine.

The amend guard has a limit. It knows a commit was pushed only from your clone’s remote-tracking branches (origin/...), so it is as current as your last fetch. And it guards --amend only: a rebase followed by git push --force rewrites shared history without asking it.

Two different questions, two commands.

Do the checks pass? - pre-commit run --all-files runs every check now, on every file. It says nothing about whether Git will run them on your next commit.
Will Git run them? - head -2 .git/hooks/pre-commit shows # File generated by pre-commit when the hook is installed. A fresh clone has none until you run make install-hooks.

Every Hook Again in CI

On GitHub the same checks run on every pull request, from the same .pre-commit-config.yaml - whether or not anyone installed the hooks. The rules are the same; the scope differs: a hook checks the files you stage, CI checks every file of the pull request’s commit.

Workflow (check name)	Runs on	What it adds
Pre-commit (`hooks`)	every pull request and push to `main`	every hook on every file - what `--no-verify` skipped. The branch guard is left out: CI checks out a single commit, and `main` is protected by the ruleset instead
Secrets guard (`secrets`, `gitleaks`)	every pull request and push to `main`	`no-secrets` and `sops-encrypted` (Chapter 04), as a separate check; `gitleaks` scans the content of the whole history (Chapter 21) - advisory, not a required check
Flux Diff (`validate`)	every pull request	builds every Kustomization and validates it against the Kubernetes and Flux schemas - kubeconform is required here, optional on your machine
Terraform - Hetzner (`terraform-plan-gate`)	every pull request; the plan itself only when the Hetzner Terraform changed	`terraform validate` and a plan of the Hetzner module. The gate is green without a Terraform change or with a successful plan, red when the plan failed or was cancelled

The Pre-commit workflow

Show pre-commit.yml

name: Pre-commit

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  hooks:
    runs-on: ubuntu-26.04
    timeout-minutes: 20
    steps:
      - name: Checkout
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Setup Python
        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.15"

      # The versions the Makefile pins for the workstation (TERRAFORM_VERSION, KUBECTL_VERSION).
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: "1.16.5"
          terraform_wrapper: false

      # kubectl, pinned and checked against the release's sha256 (dl.k8s.io/.../kubectl.sha256) - like Helm.
      - name: Setup kubectl
        env:
          KUBECTL_VERSION: "v1.37.1"
          KUBECTL_SHA256: "65691ff77eb6fa44c908b77a1082c9f092c3b9733b5cefabec0d1104890e21a8"
        run: |
          curl -sSLfO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl"
          echo "${KUBECTL_SHA256}  kubectl" | sha256sum -c -
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

      # flux build for check-app-security.sh: it builds every Flux Kustomization the way the
      # kustomize-controller does (no cluster). Pinned to the platform's Flux release, as in flux-diff.yml.
      - name: Setup Flux CLI
        uses: fluxcd/flux2/action@b9c17924adb533d3617ec7de97a7b0f4acc07b39 # v2.9.6
        with:
          version: 2.9.6   # without it the action installs the latest Flux CLI

      # Helm for the tool-versions-test hook: it asks the pinned charts what they install (helm show).
      # Pinned and checked against the release's sha256sum.
      - name: Setup Helm
        env:
          # renovate: datasource=github-releases depName=helm/helm
          HELM_VERSION: "v4.3.0"
          HELM_SHA256: "86584a54def73570558f66f5111cc53dfed56689637ae32c1201205d494f54fb"
        run: |
          archive="helm-${HELM_VERSION}-linux-amd64.tar.gz"
          curl -sSLfO "https://get.helm.sh/${archive}"
          echo "${HELM_SHA256}  ${archive}" | sha256sum -c -
          tar xzf "${archive}" --strip-components=1 linux-amd64/helm
          sudo mv helm /usr/local/bin/ && rm "${archive}"
          helm version --short

      # The Kyverno CLI for the kyverno-policies-test hook (tests/kyverno-policies.test.sh says why
      # >= 1.19). Pinned and checked against the release's checksums.txt.
      - name: Setup Kyverno CLI
        env:
          # renovate: datasource=github-releases depName=kyverno/kyverno
          KYVERNO_VERSION: "1.19.1"
          KYVERNO_SHA256: "b38228f367fc0fdc2b08f4c83ea50ac5f16c60ff8d62d76a66157c33c47b70ae"
        run: |
          archive="kyverno-cli_v${KYVERNO_VERSION}_linux_x86_64.tar.gz"
          curl -sSLfO "https://github.com/kyverno/kyverno/releases/download/v${KYVERNO_VERSION}/${archive}"
          echo "${KYVERNO_SHA256}  ${archive}" | sha256sum -c -
          tar xzf "${archive}" kyverno && sudo mv kyverno /usr/local/bin/ && rm "${archive}"
          kyverno version

      # promtool for the platform-alerts-test hook - the version the clusters' Prometheus runs.
      # Pinned and checked against the release's sha256sums.txt.
      - name: Setup promtool
        env:
          # renovate: datasource=github-releases depName=prometheus/prometheus
          PROMETHEUS_VERSION: "3.15.0"
          PROMETHEUS_SHA256: "2a542df32eac02ee17b9d844fb2aa1de00dafa5476579ba8a3ba862e9d572ea0"
        run: |
          archive="prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz"
          curl -sSLfO "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/${archive}"
          echo "${PROMETHEUS_SHA256}  ${archive}" | sha256sum -c -
          tar xzf "${archive}" --strip-components=1 "prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool"
          sudo mv promtool /usr/local/bin/ && rm "${archive}"
          promtool --version

      # checkov: the terraform-security hook fails without it (it is a required tool, not optional).
      - name: Install pre-commit and checkov
        run: pip install pre-commit==4.6.2 checkov==3.3.27

      - name: Run every hook on every file
        env:
          SKIP: master-branch-check
          # The credential dates are SafeOps' own: a learner's fork cannot rotate them, and its pull
          # requests must not fail the day one of them is due. There they are reported, not enforced.
          CHECK_CREDENTIALS_DATES: ${{ github.repository == 'safeops-course/sre' && 'enforce' || 'report' }}
        run: pre-commit run --all-files --show-diff-on-failure

Required Checks Decide the Merge

GitHub does not wait for every check. The platform’s ruleset on main (protect main) decides what a merge needs:

a pull request - nobody pushes to main directly;
the required checks passed: hooks, secrets, validate (Flux Diff) and terraform-plan-gate;
no force-push, no deletion of main, signed commits.

A check can be required only if it runs on every pull request - a required check that never starts leaves a pull request waiting forever. That is why Flux Diff has no path filter, and why the Terraform plan reports through a gate that always runs. Every other check - such as the CodeRabbit review - is advisory: red, cancelled or missing, it does not stop the button. Before you merge, read the checks list yourself: every check must have run, and a grey Cancelled or Skipped needs a reason, not a re-run. The rules of the ruleset can be bypassed by repository admins when they merge a pull request, and by the deploy key Flux image automation uses to push - nobody else.

CodeRabbit reviews pull requests (not drafts) and comments on them - a second pair of eyes that knows the repository’s rules. It is not a required check: its review advises, a person decides.

GitHub’s Own Guardrails

Some protection is not a check at all - it is a setting of the repository, so no flag on a laptop skips it:

Secret scanning with push protection: GitHub refuses a push that contains a token in a format it recognises (Chapter 04).
Signed commits on main: GitHub requires a verified signature on every commit - proof that the commit was signed by a key registered to a GitHub account, not necessarily by its author.
Actions pinned to a commit SHA: a workflow may use an action only by its full commit SHA. A tag such as v5 can be moved to other code; a SHA cannot.
Read-only workflow tokens by default: a workflow writes only where it asks for it (Flux Diff may comment on its pull request).
The environment production: an apply waits for one of its reviewers.

Terraform: Merging Only Plans

Flux delivers flux/**: after the merge, the Kustomizations apply what is on main (Chapter 03). The Hetzner infrastructure is different - a Terraform apply changes servers, networks and load balancers, and some changes cannot be undone. So a merge never applies:

The pull request gets a plan: does the change plan cleanly, and what would it do?
The merge to main gets another plan, and ends. Its summary says how to apply.
Applying is a separate, deliberate run: Actions → Terraform - Hetzner → Run workflow on main, with apply ticked. It makes a fresh plan and writes its Plan: line, the full plan, the file’s SHA-256 and the time into the run’s summary.
A person approves it in the environment production - only the environment’s Required reviewers make the run wait for a person. Repository admins can bypass that protection (the environment allows it), so the approval is the rule, not a guarantee.
The apply job applies only the file with that SHA-256, and only within 60 minutes of the plan (Chapter 02).

Why a separate run, and not an apply after every merge: not every merge should change the infrastructure now, and an apply that waits for a person holds the queue every Terraform workflow shares - the plans, the destroy, the end-to-end check. Until October 2026 every merge asked for an apply, and one approval nobody gave held that queue for six days. Now a run waits only when someone asked to apply; if it is forgotten, the person who started it cancels it. A waiting run is never approved just to make the queue move.

Terraform - Hetzner workflow

Show terraform-hcloud.yml

name: Terraform - Hetzner

on:
  pull_request:
  push:
    branches: [main]
    paths:
      - "infra/terraform/hcloud_cluster/**"
      - ".github/workflows/terraform-hcloud*.yml"
  # Manual run from main: a plan, and with "apply" a fresh plan that is applied after approval.
  workflow_dispatch:
    inputs:
      apply:
        description: "Apply the plan (after approval in the environment production)"
        type: boolean
        default: false

env:
  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"

permissions:
  contents: read

jobs:
  # Does this run concern the Hetzner Terraform? Pushes to main come here only through the path filter
  # above and manual runs always do; a pull request does when it changes these paths.
  changes:
    runs-on: ubuntu-26.04
    outputs:
      terraform: ${{ steps.diff.outputs.terraform }}
    steps:
      - name: Checkout
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0

      - name: Changed paths
        id: diff
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
        run: |
          if [ "$GITHUB_EVENT_NAME" != "pull_request" ]; then
            echo "terraform=true" >> "$GITHUB_OUTPUT"
            exit 0
          fi
          # A failing git diff must not read as "nothing changed": check it on its own.
          if ! changed="$(git diff --name-only "$BASE_SHA"...HEAD)"; then
            echo "::error::git diff against the base commit failed"
            exit 1
          fi
          if grep -qE '^(infra/terraform/hcloud_cluster/|\.github/workflows/terraform-hcloud)' <<< "$changed"; then
            echo "terraform=true" >> "$GITHUB_OUTPUT"
          else
            echo "terraform=false" >> "$GITHUB_OUTPUT"
          fi

  plan:
    needs: changes
    if: needs.changes.outputs.terraform == 'true'
    runs-on: ubuntu-26.04
    # Job-level, not workflow-level: a pull request without Terraform changes must not wait in the
    # queue that the plans, the applies, the destroy and the e2e check share (never two runs on the
    # same state).
    concurrency:
      group: terraform-hcloud
      cancel-in-progress: false
      # queue: max - runs wait in order instead of the older pending one being cancelled (the default
      # keeps one pending run and cancels the rest - that is how PR plans were lost in Sep 2026).
      queue: max
    env:
      # "true" only for a manual run from main with "apply" ticked - the only run that applies.
      APPLY: ${{ github.event_name == 'workflow_dispatch' && inputs.apply && github.ref == 'refs/heads/main' }}
    outputs:
      has_changes: ${{ steps.plan.outputs.has_changes }}
      plan_sha256: ${{ steps.fingerprint.outputs.plan_sha256 }}
      plan_created: ${{ steps.fingerprint.outputs.plan_created }}

    defaults:
      run:
        working-directory: infra/terraform/hcloud_cluster

    steps:
      - name: Checkout
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Setup kubectl
        run: |
          curl -sLO "https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/kubectl"
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: "1.16.5"
          terraform_wrapper: false

      - name: Terraform fmt
        run: terraform fmt -check -recursive

      - name: Load the node SSH key into ssh-agent
        env:
          NODE_SSH_KEY: ${{ secrets.HCLOUD_SSH_PRIVATE_KEY }}
        run: |
          # kube-hetzner signs in to the nodes through ssh-agent (ssh_private_key = null), so the
          # private key is never a Terraform variable - neither in the saved plan nor in the state.
          eval "$(ssh-agent -s)" > /dev/null
          printf '%s\n' "$NODE_SSH_KEY" | ssh-add -
          echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV"
          echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV"

      - name: Terraform init
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: terraform init -input=false

      - name: Terraform validate
        run: terraform validate

      - name: Terraform plan
        id: plan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
          TF_VAR_hcloud_token: ${{ secrets.HCLOUD_TOKEN }}
          TF_VAR_ssh_public_key: ${{ vars.HCLOUD_SSH_PUBLIC_KEY }}
          TF_VAR_flux_git_repository_url: https://github.com/${{ github.repository }}.git
          TF_VAR_flux_git_repository_branch: main
          TF_VAR_flux_kustomization_path: ./flux/bootstrap/profiles/hetzner
          TF_VAR_sops_age_key: ${{ secrets.SOPS_AGE_KEY }}
          # The OpenTelemetry collector's export target (cluster-secrets). Sensitive, not ephemeral: it is
          # in the saved - encrypted - plan, so the apply step does not need it again.
          TF_VAR_uptrace_dsn: ${{ secrets.UPTRACE_DSN }}
          TF_VAR_backup_s3_access_key_id: ${{ secrets.BACKUP_S3_ACCESS_KEY_ID }}
          TF_VAR_backup_s3_secret_access_key: ${{ secrets.BACKUP_S3_SECRET_ACCESS_KEY }}
          TF_VAR_backup_s3_bucket: ${{ vars.BACKUP_S3_BUCKET }}
          TF_VAR_backup_s3_endpoint: ${{ vars.BACKUP_S3_ENDPOINT }}
          TF_VAR_backup_s3_region: ${{ vars.BACKUP_S3_REGION }}
        run: |
          set +e
          set -o pipefail
          terraform plan -input=false -lock-timeout=5m -no-color -detailed-exitcode -out=tfplan 2>&1 | tee plan.txt
          exit_code=${PIPESTATUS[0]}
          set -e
          if [ "$exit_code" -eq 1 ]; then
            echo "Terraform plan failed."
            exit 1
          fi
          if [ "$exit_code" -eq 0 ]; then
            echo "has_changes=false" >> "$GITHUB_OUTPUT"
          else
            echo "has_changes=true" >> "$GITHUB_OUTPUT"
          fi

      # What the approver reads is what gets applied: the plan text, the SHA-256 of the plan file
      # and the time it was made go to the run's summary - the page with "Review deployments".
      # The apply job refuses a file with another hash, or a plan older than PLAN_MAX_AGE_MINUTES.
      - name: Fingerprint the plan
        id: fingerprint
        if: env.APPLY == 'true' && steps.plan.outputs.has_changes == 'true'
        run: |
          echo "plan_sha256=$(sha256sum tfplan | cut -d' ' -f1)" >> "$GITHUB_OUTPUT"
          echo "plan_created=$(date +%s)" >> "$GITHUB_OUTPUT"

      - name: Plan summary
        if: always() && steps.plan.outcome != 'skipped'
        env:
          PLAN_SHA256: ${{ steps.fingerprint.outputs.plan_sha256 }}
          PLAN_MAX_AGE_MINUTES: "60"
        run: |
          # The approver must see the plan: a missing plan.txt or a summary GitHub will not show
          # (over 1 MiB per step) fails here, before the plan is encrypted and uploaded.
          if [ ! -s plan.txt ]; then
            echo "::error::plan.txt is missing or empty - there is no plan to show the approver"
            exit 1
          fi
          summary="$(mktemp)"
          {
            echo "## Terraform plan (Hetzner)"
            echo
            echo "**$(grep -E '^(Plan:|No changes)' plan.txt | tail -1)**"
            if [ -n "$PLAN_SHA256" ]; then
              echo
              echo "- plan file SHA-256: \`$PLAN_SHA256\` - the apply job applies only a file with this hash"
              echo "- made at $(date -u '+%Y-%m-%d %H:%M UTC'); the apply refuses it after ${PLAN_MAX_AGE_MINUTES} minutes - then re-run the workflow for a fresh plan"
              echo "- approve only what you read below: this is the plan that will run, not the one from the pull request"
            elif [ "$GITHUB_EVENT_NAME" != "pull_request" ] && grep -q '^Plan:' plan.txt; then
              echo
              echo "- **not applied** - this run only plans. To apply: Actions > Terraform - Hetzner > Run workflow (branch main), tick \"apply\"; that makes a fresh plan and waits for approval"
            fi
            echo
            echo "<details><summary>Full plan</summary>"
            echo
            echo '```'
            cat plan.txt
            echo '```'
            echo "</details>"
          } > "$summary"
          size="$(wc -c < "$summary")"
          if [ "$size" -gt 1000000 ]; then
            echo "::error::the plan summary is ${size} bytes; GitHub shows at most 1 MiB per step, so the approver could not read the plan - split the change"
            exit 1
          fi
          cat "$summary" >> "$GITHUB_STEP_SUMMARY"

      # A saved plan holds every non-ephemeral input variable in plain text - the Hetzner token
      # and the backup keys (the SOPS age key is ephemeral, the node SSH key goes through
      # ssh-agent) - and artifacts of a public repository can be downloaded by any GitHub user.
      # Only the encrypted file leaves this job; the apply job decrypts it and replays the plan.
      - name: Encrypt tfplan
        if: env.APPLY == 'true' && steps.plan.outputs.has_changes == 'true'
        env:
          TFPLAN_PASSPHRASE: ${{ secrets.TFPLAN_PASSPHRASE }}
        run: |
          if [ -z "$TFPLAN_PASSPHRASE" ]; then
            echo "::error::secret TFPLAN_PASSPHRASE is not set - the plan is never uploaded unencrypted"
            exit 1
          fi
          openssl enc -aes-256-cbc -pbkdf2 -iter 600000 -salt -pass env:TFPLAN_PASSPHRASE -in tfplan -out tfplan.enc
          rm -f tfplan

      - name: Upload encrypted tfplan
        if: env.APPLY == 'true' && steps.plan.outputs.has_changes == 'true'
        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
        with:
          # the attempt in the name: a re-run downloads its own plan, never an earlier attempt's
          name: terraform-hcloud-tfplan-${{ github.run_attempt }}
          path: |
            infra/terraform/hcloud_cluster/tfplan.enc
            infra/terraform/hcloud_cluster/plan.txt
          if-no-files-found: error
          retention-days: 1

  apply:
    runs-on: ubuntu-26.04
    needs: plan
    if: github.event_name == 'workflow_dispatch' && inputs.apply && github.ref == 'refs/heads/main' && needs.plan.outputs.has_changes == 'true'
    # Approval: the "production" environment requires a reviewer (Settings > Environments). The job
    # waits in the run page ("Review deployments") - no issue, no third-party action with a token.
    environment: production
    permissions:
      contents: read
      actions: read # gh run download of the plan artifact
    concurrency:
      group: terraform-hcloud
      cancel-in-progress: false
      # queue: max - runs wait in order instead of the older pending one being cancelled (the default
      # keeps one pending run and cancels the rest - that is how PR plans were lost in Sep 2026).
      queue: max

    defaults:
      run:
        working-directory: infra/terraform/hcloud_cluster

    steps:
      - name: Checkout
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Setup kubectl
        run: |
          curl -sLO "https://dl.k8s.io/release/v1.37.1/bin/linux/amd64/kubectl"
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
        with:
          terraform_version: "1.16.5"

      - name: Load the node SSH key into ssh-agent
        env:
          NODE_SSH_KEY: ${{ secrets.HCLOUD_SSH_PRIVATE_KEY }}
        run: |
          # kube-hetzner signs in to the nodes through ssh-agent (ssh_private_key = null), so the
          # private key is never a Terraform variable - neither in the saved plan nor in the state.
          eval "$(ssh-agent -s)" > /dev/null
          printf '%s\n' "$NODE_SSH_KEY" | ssh-add -
          echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV"
          echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV"

      - name: Terraform init
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: terraform init -input=false

      # gh run download, not actions/download-artifact: its latest release (v8.0.2) prints a Node
      # DEP0005 (Buffer) deprecation warning on every run. gh is on the runner; the token reads only
      # this repository's Actions artifacts (permissions: actions: read).
      - name: Download tfplan artifact
        env:
          GH_TOKEN: ${{ github.token }}
        run: gh run download "${GITHUB_RUN_ID}" --repo "${GITHUB_REPOSITORY}" --name "terraform-hcloud-tfplan-${GITHUB_RUN_ATTEMPT}" --dir .

      - name: Decrypt tfplan
        env:
          TFPLAN_PASSPHRASE: ${{ secrets.TFPLAN_PASSPHRASE }}
        run: |
          openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 -pass env:TFPLAN_PASSPHRASE -in tfplan.enc -out tfplan
          rm -f tfplan.enc

      - name: Apply only the reviewed plan, and only while it is fresh
        env:
          PLAN_SHA256: ${{ needs.plan.outputs.plan_sha256 }}
          PLAN_CREATED: ${{ needs.plan.outputs.plan_created }}
          PLAN_MAX_AGE_MINUTES: "60"
        run: |
          if [ -z "$PLAN_SHA256" ] || [ -z "$PLAN_CREATED" ]; then
            echo "::error::the plan job reported no hash or time - refusing to apply an unidentified plan"
            exit 1
          fi
          actual="$(sha256sum tfplan | cut -d' ' -f1)"
          if [ "$actual" != "$PLAN_SHA256" ]; then
            echo "::error::this plan file ($actual) is not the one shown in the run summary ($PLAN_SHA256)"
            exit 1
          fi
          age_seconds=$(( $(date +%s) - PLAN_CREATED ))
          if [ "$age_seconds" -gt $(( PLAN_MAX_AGE_MINUTES * 60 )) ]; then
            echo "::error::the plan is ${age_seconds}s old (limit ${PLAN_MAX_AGE_MINUTES} minutes); reality may have moved under it - re-run the workflow for a fresh plan and review it again"
            exit 1
          fi
          echo "applying exactly the reviewed plan $actual (${age_seconds}s old)"

      - name: Terraform apply
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
          # ephemeral: not in the saved plan, so the apply must be given it again
          TF_VAR_sops_age_key: ${{ secrets.SOPS_AGE_KEY }}
        run: terraform apply -input=false -lock-timeout=5m tfplan

  # The required check of a pull request (add "terraform-plan-gate" to the ruleset on main). It runs
  # on every pull request: green when the pull request does not touch the Hetzner Terraform, or when
  # its plan succeeded; red when the plan failed, was cancelled or did not run although it should.
  # A pull request from a fork gets no secrets, so a fork's Terraform change fails here.
  terraform-plan-gate:
    needs: [changes, plan]
    if: always() && github.event_name == 'pull_request'
    runs-on: ubuntu-26.04
    steps:
      - name: Plan result
        env:
          CHANGES: ${{ needs.changes.result }}
          TERRAFORM: ${{ needs.changes.outputs.terraform }}
          PLAN: ${{ needs.plan.result }}
        run: |
          if [ "$CHANGES" != "success" ]; then
            echo "::error::could not tell whether this pull request changes the Hetzner Terraform (changes: $CHANGES)"
            exit 1
          fi
          if [ "$TERRAFORM" != "true" ]; then
            echo "No Hetzner Terraform change - nothing to plan."
            exit 0
          fi
          if [ "$PLAN" != "success" ]; then
            echo "::error::the Terraform plan of this pull request did not succeed (plan: $PLAN)"
            exit 1
          fi
          echo "The Terraform plan of this pull request succeeded."

Your Fork Has None of This - Until You Add It

Your fork has the hooks and the workflow files, but not the platform’s settings:

GitHub Actions - may be disabled in a fork. Enable them in your fork’s Actions tab - the lab needs them, and the Pre-commit, Secrets guard and Flux Diff checks run on your pull requests. They need no secrets. The Terraform workflows do - in your fork they fail, so ignore them or disable them there.
No ruleset - your main takes any push, and a pull request with a red check still merges: a check that does not block is only advice. The lab adds one first (A5): a pull request and the checks hooks and secrets required.
No CodeRabbit - it reviews only repositories that are connected to it; connecting yours is optional.

This chapter’s lab enables Actions and adds the ruleset - both are required there. Beyond the lab, keeping them is your choice; without them your fork has only the hooks - which one flag skips.

From now on your own changes go the same way: a branch, a pull request in your fork, a merge - and Flux applies what reaches main.

Safe Workflow (Step-by-Step)

Hooks installed: make install-hooks once per clone; head -2 .git/hooks/pre-commit proves it.
Work on a branch: git switch -c <topic>. Commit without --no-verify; when a hook fails, fix the cause.
Push and open a pull request. Wait until every check has run.
Read the checks: required ones passed; advisory ones read - a red Flux Diff or plan is a finding, a cancelled one needs its reason.
Merge. Flux applies flux/**. For the Hetzner Terraform, the merge only plans; applying is a separate run that a person approves after reading the fresh plan.

Instructions for Your AI Agent

The rules this chapter adds:

Never git commit --no-verify or git push --no-verify; never amend or force-push a pushed commit. Enforced by: the amend guard (prevent-amend-after-push), which runs even with --no-verify; the Pre-commit CI job, which runs the skipped checks on the pull request; for a push to main, the pre-push branch guard and the ruleset on the platform’s main - and on your fork’s, from the lab on.
Report a pull request as ready only when every check ran and passed. A cancelled, skipped or missing check is not green - say which one and why. Enforced by: the required checks block the merge until they pass - failed, cancelled or never started. GitHub counts a job skipped by its if: as passed; the agent still reports the skip when the change should have run it. Every other check - instruction only.
Never start a Terraform apply (Run workflow with “apply”), and never approve or reject a deployment. Enforced by: the environment production requires a person’s approval; starting the run - instruction only.
Never cancel or re-run someone else’s workflow run, even one that blocks the queue - report it with its link. Enforced by: instruction only.
Never change rulesets, required checks, environments or repository settings. Enforced by: repository admin rights; for an agent that has them - instruction only.

The full list, chapter by chapter, is docs/agent-rules.md in your fork of the platform repository.

This builds on: Secrets Management (Chapter 04) - sops-encrypted is one of the hooks that now runs twice: on your machine and on every pull request. This enables: Network Policies (Chapter 06) - every policy from now on reaches the cluster through a pull request whose checks you can read.