Core Track Guardrails-first chapter in core learning path.

Estimated Time

  • Reading: 35 min
  • Lab: 50 min
  • Quiz: 5 min

Prerequisites

What You Will Produce

On your own kind cluster: proof that two copies of the same code with their own local state create the same objects twice - once with an error, once silently - and that a shared state with a lock (MinIO) gives them one memory and stops a second writer. Then proof that a saved plan silently overwrites a change made by hand, and that a fresh plan and the guard’s age limit catch it. It also shows that a password Terraform calls (sensitive value) sits in the state and the saved plan in plain text. The lab ends where it started: terraform plan -detailed-exitcode returns 0.

Chapter 02: Infrastructure as Code with Terraform

Interactive Explainer

Video is being produced. Check back soon.

Learning Objectives

By the end of this chapter, you will be able to:

  • Explain what Terraform’s state is and how a plan is made from the code, the state and the real objects
  • Show why two independent states for one infrastructure break it - two people or jobs, each with their own local state - and move a state to a shared backend with locking
  • Apply only the plan you have read, while it is fresh - with guard-terraform-plan.sh on a workstation, and with a hash and an age check in CI
  • Detect drift with terraform plan -detailed-exitcode and decide whether to revert it or put it into code
  • Explain why sensitive does not protect a value in the saved plan or the state, and keep secrets out of both
  • Explain what terraform destroy cannot see in a cloud cluster, and how to prove that nothing is left

Recall Check

Before continuing, quickly recall:

  • The engineer’s kubectl patch was the right command - why did it change production? (Chapter 01)
  • What does rule II, plan before apply, ask for? (Chapter 01)
  • Which three commands prove that your cluster is green? (Chapter 00)

If you can’t answer these, revisit the corresponding chapter before proceeding.

Context

In Chapter 00 one terraform apply built your cluster. From then on, Terraform owns what it built: it keeps a record of every object in its state and changes them only when you apply a plan. That record is Terraform’s memory, and this chapter is about keeping it right: one shared, locked memory for everyone who applies, only the plan someone has actually read, and what the memory and a saved plan contain - every value Terraform writes, secrets included.

The chapter has two parts. The lab runs on your kind cluster: it starts with a local state file, then gives two copies of a small module one shared state in MinIO (running in the cluster) and shows the lock stopping two applies at once. The incident and the platform’s own examples come from Hetzner Cloud - the shared backend in Cloudflare R2, the CI workflow that applies only an approved plan, and the destroy - because that is where the stakes are real: servers billed by the hour, and a team that applies from several laptops and from CI. The cause is not the cloud - it is several writers with different memories of the same infrastructure, which the lab reproduces on your own machine. You read them; you do not need a cloud account (they are the Cloud track).

Work through each lesson section; the chapter video, once it is published, gives the concept overview.

What Terraform Builds

Diagram: terraform apply infra/terraform/kind_cluster; kind cluster sre-control-plane - containers in Docker; control-plane node localhost:8080 → 30080 localhost:8443 → 30443; worker node; metrics-server; Flux Operator + FluxInstance follows your fork; develop · staging · production · observability namespaces cluster-config ConfigMap secrets and your SOPS key; everything else: the ingress (Traefik), apps, databases, policies, observability; from here on Git drives it (Chapter 03)

Terraform builds the cluster and hands it to Flux. What Terraform created, Terraform owns - change it through a plan. Everything Flux deploys from Git is covered in Chapter 03.

Four words first. A provider is a plugin that lets Terraform talk to one system’s API and gives it the resource types for that system - the kind provider for kind clusters, kubernetes and helm for what runs in a cluster, hcloud for Hetzner Cloud. A resource is one object that a provider creates and Terraform then owns, written in the code as resource "<type>" "<name>". Its address is <type>.<name>: kind_cluster.sre is the resource of type kind_cluster (from the kind provider) that the code names sre - the cluster it creates is called sre-control-plane. The plan, the state and the rest of the code all refer to it by that address. A reference uses a value of one resource in another - kind_cluster.sre.endpoint is the address of the cluster’s API - and tells Terraform which resource has to exist first.

How the module works. You describe the end state, and Terraform works out the order from the references between resources:

  1. kind_cluster.sre creates the two Docker containers that act as nodes and returns the cluster’s address and client certificate.
  2. The Kubernetes and Helm providers are configured from those values (references such as kind_cluster.sre.endpoint and kind_cluster.sre.client_certificate), so every later step talks to exactly this cluster - never to whatever your kubectl context points at.
  3. A Helm release installs metrics-server. The ingress is not here: Traefik, on ports 8080/8443, comes from Git through Flux - on this cluster and on every other. A cluster module gives only what the provider has to - here the nodes and their ports; on Hetzner (Cloud track) also the load balancer.
  4. Two steps install the Flux Operator and a FluxInstance that follows your fork’s Git repository.
  5. Terraform also creates the environment namespaces, the cluster-config ConfigMap and the secrets the apps need - the objects Flux expects to find when it starts deploying. The passwords come from random_password resources, so they are generated once and then stay the same.

Production note: configuring the Kubernetes and Helm providers from a cluster created in the same apply (step 2) is a convenience of this lab. The Kubernetes provider documentation warns against it: Terraform configures providers before it creates resources, which leads to “intermittent and unpredictable errors” - for example when the cluster does not exist yet, is replaced or gets new certificates. It recommends managing the cluster and what runs in it with separate apply operations. In production the cluster is its own Terraform layer with its own state, and what runs inside it comes from a second layer or - as in this course from Chapter 03 on - from GitOps.

The principle: the cluster is code, not a sequence of commands. Anyone with the repository gets the same cluster from one terraform apply, a plan shows every change before it happens, and the state records what exists. The code alone is not enough, though: Terraform needs its memory of what it built, and the rest of this chapter is about keeping that memory right.

Kind cluster module

Show the Terraform code
  required_providers {
    kind = {
      source  = "tehcyx/kind"
      version = "~> 0.11"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 3.3"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 3.2"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.2"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
    time = {
      source  = "hashicorp/time"
      version = "~> 0.14"
    }
  }
}

resource "terraform_data" "minio_data_dir" {
  input = local.minio_data_dir

  provisioner "local-exec" {
    command = "mkdir -p \"${local.minio_data_dir}\" && chmod 0777 \"${local.minio_data_dir}\""
  }
}

  name = "sre-control-plane"
  # kind's own wait waits for Ready nodes, and a node is Ready only once a CNI runs - ours (Cilium) is
  # installed right after, so waiting here would never end. helm_release.cilium waits for it instead.
  wait_for_ready  = false
  kubeconfig_path = local.kubeconfig_path
  node_image      = var.kind_node_image

  kind_config {
    api_version = "kind.x-k8s.io/v1alpha4"
    kind        = "Cluster"

    # Dual-stack: every pod and Service gets an IPv4 and an IPv6 address, as on the platform's dual-stack
    # cloud clusters (a provider gets IPv6 where it has it by default - docs/providers.md). kind's
    # own CNI (kindnet) is off - Cilium is installed below, the policy engine the clouds run too.
    # The IPv6 ranges are unique local (fd00::/8, not routed on the internet) - kind's own defaults.
    networking {
      api_server_address  = "127.0.0.1"
      api_server_port     = 6443
      kube_proxy_mode     = "iptables"
      ip_family           = "dual"
      pod_subnet          = "10.244.0.0/16,fd00:10:244::/56"
      service_subnet      = "10.96.0.0/16,fd00:10:96::/112"
      disable_default_cni = true
    }

    containerd_config_patches = [
      <<-EOT
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5001"]
          endpoint = ["http://kind-registry:5000"]
      EOT
    ]

    node {
      role = "control-plane"

      kubeadm_config_patches = [
        <<-EOT
          kind: InitConfiguration
          nodeRegistration:
            kubeletExtraArgs:
              node-labels: "ingress-ready=true"
              authorization-mode: "Webhook"
        EOT
      ]

      extra_port_mappings {
        container_port = 30080
        host_port      = 8080
        listen_address = "127.0.0.1"
        protocol       = "TCP"
      }

      extra_port_mappings {
        container_port = 30443
        host_port      = 8443
        listen_address = "127.0.0.1"
        protocol       = "TCP"
      }
    }

    node {
      role = "worker"

      # MinIO's PersistentVolume (flux/infrastructure/data/minio/pvc.yaml) points at this path.
      extra_mounts {
        host_path      = local.minio_data_dir
        container_path = "/var/local/minio-data"
      }
    }
  }
}

  provisioner "local-exec" {
    when        = create
    command     = "${path.module}/scripts/merge-kubeconfig.sh \"${local.kubeconfig_path}\""
    interpreter = ["/bin/bash", "-c"]
  }

  # A new kind cluster (for example a new node_image) needs this step again;
  # without it the replaced cluster came up without Flux.
  lifecycle {
    replace_triggered_by = [kind_cluster.sre]
  }
}

  provisioner "local-exec" {
    when        = create
    interpreter = ["/bin/bash", "-c"]
    command     = "kubectl --kubeconfig=\"${local.kubeconfig_path}\" -n kube-system rollout status ds/cilium --timeout=300s && kubectl --kubeconfig=\"${local.kubeconfig_path}\" wait --for=condition=Ready nodes --all --timeout=300s || { kubectl --kubeconfig=\"${local.kubeconfig_path}\" get nodes; kubectl --kubeconfig=\"${local.kubeconfig_path}\" -n kube-system get pods -l k8s-app=cilium; exit 1; }"
  }

  lifecycle {
    replace_triggered_by = [kind_cluster.sre, helm_release.cilium]
  }
}

  # A new kind cluster (for example a new node_image) needs this step again;
  # without it the replaced cluster came up without Flux.
  lifecycle {
    replace_triggered_by = [kind_cluster.sre]
  }
}

output "kubeconfig_load_instructions" {
  description = "How to use the generated kubeconfig"
  value       = <<-EOT
    # The kubeconfig is already merged into ~/.kube/config. Name the cluster in every
    # command instead of switching the shared current context (Chapter 01):
    kubectl --context kind-sre-control-plane get nodes
    flux --context kind-sre-control-plane get kustomizations -A
  EOT
}

removed {
  from = kubernetes_namespace_v1.traefik
  lifecycle { destroy = false }
}

  depends_on = [null_resource.merge_kubeconfig]

  # A new kind cluster (a new node_image, a networking change) has no CNI: install Cilium there again,
  # or its nodes stay NotReady and nothing after this ever starts.
  lifecycle {
    replace_triggered_by = [kind_cluster.sre]
  }

  values = [yamlencode({
    ipam                 = { mode = "kubernetes" }
    k8s                  = { requireIPv4PodCIDR = true, requireIPv6PodCIDR = true }
    ipv4                 = { enabled = true }
    ipv6                 = { enabled = true }
    kubeProxyReplacement = false
    routingMode          = "tunnel"
    endpointRoutes       = { enabled = true }
    bpf                  = { masquerade = false }
    rollOutCiliumPods    = true
    operator             = { replicas = 1 }
    hubble               = { enabled = false }
    # An ipBlock matches node addresses too, as in Kubernetes' own model: Cilium leaves nodes out of CIDR
    # rules by default, and the API server runs on a node here (kind, k3s) - without this, Postgres'
    # instance manager (allow-postgres-egress-apiserver) could not reach it. cluster-conformance C3 checks.
    policyCIDRMatchMode = "nodes"
    image               = { pullPolicy = "IfNotPresent" }
  })]
}

  depends_on = [time_sleep.wait_for_cluster]

  set = [
    {
      name  = "args[0]"
      value = "--kubelet-insecure-tls"
    },
  ]
}

  # Only what the operator install depends on (as in gke and hcloud); the repository, branch and path are
  # the FluxInstance's inputs and re-create only null_resource.flux_instance below.
  triggers = {
    kubeconfig_path = local.kubeconfig_path
    # reinstall when the pinned Flux Operator release changes
    operator_version = var.flux_operator_version
    # a re-pinned manifest (same version) is installed and checked again
    manifest_sha256 = local.flux_operator_manifest_sha256
  }

  provisioner "local-exec" {
    when        = create
    interpreter = ["/bin/bash", "-c"]
    command     = "\"${path.module}/../scripts/flux-operator-install.sh\" \"${local.kubeconfig_path}\" \"${var.flux_operator_version}\" \"${local.flux_operator_manifest_sha256}\""
  }

  # A new kind cluster (for example a new node_image) needs this step again;
  # without it the replaced cluster came up without Flux.
  lifecycle {
    precondition {
      condition     = local.flux_operator_manifest_sha256 != ""
      error_message = "Flux Operator ${var.flux_operator_version} has no line in infra/terraform/scripts/flux-operator.sha256 - add its install.yaml SHA-256 (the file says how), then plan again."
    }
    replace_triggered_by = [kind_cluster.sre]
  }
}

  triggers = {
    kubeconfig_path = local.kubeconfig_path
    # Re-create the FluxInstance when its sync spec changes; before this only
    # the kubeconfig path was tracked, so switching repo/branch/path/token
    # after the first apply silently did nothing.
    repo_url    = var.flux_git_repository_url
    repo_branch = var.flux_git_repository_branch
    repo_path   = var.flux_kustomization_path
    pull_secret = local.flux_pull_secret_yaml
    # re-apply the FluxInstance when the pinned Flux version changes
    flux_version = var.flux_version
  }

  provisioner "local-exec" {
    when        = create
    command     = <<-EOC
      cat <<EOF | kubectl --kubeconfig="${local.kubeconfig_path}" apply -f -
apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: "${var.flux_version}"
    registry: ghcr.io/fluxcd
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
    - image-reflector-controller
    - image-automation-controller
  cluster:
    type: kubernetes
  sync:
    kind: GitRepository
    url: "${var.flux_git_repository_url}"
    ref: "refs/heads/${var.flux_git_repository_branch}"
    provider: generic
    path: "${var.flux_kustomization_path}"
${local.flux_pull_secret_yaml}
EOF
    EOC
    interpreter = ["/bin/bash", "-c"]
  }

  provisioner "local-exec" {
    when       = destroy
    on_failure = continue
    # Wait for the operator to finish uninstalling: with --wait=false a
    # replacement (new FluxInstance applied while the old one is still
    # terminating) gets deleted together with the old one.
    command     = "kubectl --kubeconfig=\"${self.triggers.kubeconfig_path}\" delete fluxinstance flux -n flux-system --ignore-not-found=true --wait=true --timeout=5m"
    interpreter = ["/bin/bash", "-c"]
  }

  # A new kind cluster (for example a new node_image) needs this step again;
  # without it the replaced cluster came up without Flux.
  lifecycle {
    replace_triggered_by = [kind_cluster.sre]
  }
}

  input = {
    kubeconfig_path = local.kubeconfig_path
    namespaces      = "develop,staging,production,observability,traefik,minio"
  }

  provisioner "local-exec" {
    when        = destroy
    on_failure  = continue
    command     = "\"${path.module}/../scripts/flux-pre-destroy.sh\" \"${self.input.kubeconfig_path}\" \"${self.input.namespaces}\""
    interpreter = ["/bin/bash", "-c"]
  }

  # A new kind cluster (for example a new node_image) needs this step again;
  # without it the replaced cluster came up without Flux.
  lifecycle {
    replace_triggered_by = [kind_cluster.sre]
  }
}

  lifecycle {
    destroy = false
  }
}

  metadata {
    name      = "flux-system"
    namespace = "flux-system"
  }

  data = {
    username = "git"
    password = var.flux_git_token
  }

  type = "Opaque"
}

  data = {
    cloudflare_proxied = "disabled"
    # Names the provider in the external-dns owner ID (k8s-external-dns-<cluster_name>-<cloud_provider>).
    cloud_provider = "kind"
    # A fact about the provider (IPv6 by default: dual), never read from the setting it checks: turning
    # IPv6 off must fail conformance C13, not change the promise.
    ip_stack = "dual"
    # The pods' IPv6 range (networking.pod_subnet above): NetworkPolicies cut it out of ::/0 (postBuild).
    cluster_ipv6_range    = "fd00:10:244::/56"
    cluster_name          = "sre-control-plane"
    image_registry        = var.image_registry
    git_owner             = var.git_owner
    guardian_llm_provider = var.guardian_llm_provider
  }

  depends_on = [null_resource.flux_operator_install]
}

  type = "Opaque"

  data = {
    uptrace_dsn = var.uptrace_dsn
  }

  depends_on = [null_resource.flux_operator_install]
}

  metadata {
    name = each.key
  }

  depends_on = [time_sleep.wait_for_cluster]

  lifecycle {
    # Flux owns labels (pod-security, kustomize.toolkit) and annotations
    # (kustomize.toolkit.fluxcd.io/prune) on these namespaces; Terraform only
    # guarantees they exist early enough for the generated secrets.
    ignore_changes = [
      metadata[0].labels,
      metadata[0].annotations,
    ]
  }
}

  metadata {
    name      = "ghcr-credentials-docker"
    namespace = each.key
  }

  type = "kubernetes.io/dockerconfigjson"

  data = {
    ".dockerconfigjson" = jsonencode({
      auths = {
        "ghcr.io" = {
          username = var.ghcr_username
          password = var.ghcr_token
          auth     = base64encode("${var.ghcr_username}:${var.ghcr_token}")
        }
      }
    })
  }
}


  metadata {
    name      = "sops-age"
    namespace = "flux-system"
  }

  type = "Opaque"

  data_wo = {
    "age.agekey" = var.sops_age_key != "" ? var.sops_age_key : data.local_file.age_key[0].content
  }
  data_wo_revision = var.sops_age_key_revision + (var.local_profile ? parseint(substr(sha256(data.local_file.age_key[0].content), 0, 8), 16) : 0)
}

  lifecycle {
    ignore_changes = [input]
  }
}

  metadata {
    name      = "backup-s3"
    namespace = "flux-system"
  }

  data = var.local_profile ? {
    BACKUP_S3_ENDPOINT = "http://minio.minio.svc.cluster.local:9000"
    BACKUP_S3_BUCKET   = "sre"
    BACKUP_CLUSTER_ID  = terraform_data.cluster_id.output
    } : {
    BACKUP_S3_ENDPOINT = var.backup_s3_endpoint
    BACKUP_S3_BUCKET   = var.backup_s3_bucket
    BACKUP_CLUSTER_ID  = terraform_data.cluster_id.output
  }
}

resource "kubernetes_secret_v1" "guardian" {
  count = var.local_profile ? 1 : 0

  metadata {
    name      = "k8s-ai-monitor-secrets"
    namespace = "observability"
  }

  type = "Opaque"

  data = merge(
    { "internal-token" = random_password.guardian_internal_token[0].result },
    var.guardian_llm_api_key == "" ? {} : { "${var.guardian_llm_provider}-api-key" = var.guardian_llm_api_key },
  )

  depends_on = [kubernetes_namespace_v1.bootstrap]
}

resource "kubernetes_secret_v1" "postgres_app_production" {
  count = var.local_profile ? 0 : 1

  metadata {
    name      = "app-postgres-app"
    namespace = "production"
    labels = {
      "cnpg.io/reload" = "true"
    }
  }

  type = "kubernetes.io/basic-auth"

  data = {
    username = "app"
    password = random_password.postgres_app_production[0].result
  }

  # CNPG adopts the Secret and adds connection keys and labels; Terraform only seeds it.
  lifecycle {
    ignore_changes = [data, metadata[0].labels, metadata[0].annotations]
  }

  depends_on = [kubernetes_namespace_v1.bootstrap]
}

resource "kubernetes_secret_v1" "cnpg_backup_s3" {
  for_each = local.backup_s3_secret_enabled ? toset(["develop", "staging", "production"]) : toset([])

  metadata {
    name      = "cnpg-backup-s3"
    namespace = each.key
  }

  type = "Opaque"

  data = merge(
    {
      ACCESS_KEY_ID     = var.backup_s3_access_key_id
      ACCESS_SECRET_KEY = var.backup_s3_secret_access_key
      BUCKET            = var.backup_s3_bucket
    },
    var.backup_s3_endpoint != "" ? { ENDPOINT = var.backup_s3_endpoint } : {},
    var.backup_s3_region != "" ? { REGION = var.backup_s3_region } : {},
  )

  depends_on = [kubernetes_namespace_v1.bootstrap]
}

output "flux_operator_installed" {
  description = "Indicates that Flux Operator has been installed"
  value       = null_resource.flux_operator_install.id != ""
}

output "flux_instance_created" {
  description = "Indicates that FluxInstance has been created"
  value       = "flux"
  depends_on  = [null_resource.flux_instance]
}

Detailed Lessons

Hands-On Materials

Labs, quizzes and runbooks for this chapter are free - sign in with your email, Google or GitHub to open them.

  • Lab: One Memory, and Only the Plan You Read Free · sign in
  • Quiz: Chapter 02 (Infrastructure as Code with Terraform) Free · sign in
Sign in free