Appendix: Local Development Environment

Incident Hook

A learner tests every infrastructure and GitOps change only against the cloud cluster. Feedback is slow, mistakes are expensive, and trivial YAML errors burn real time and real money. By the time the change reaches the shared environment, the debugging loop is already too wide. A local cluster exists to reduce that blast radius before cloud validation even starts.

Why This Appendix Exists

The main course teaches the production path first. This appendix shows the fastest safe feedback loop for local experimentation:

  • a Terraform-managed kind cluster
  • generated kubeconfig and context wiring
  • optional local image registry
  • Flux bootstrap for GitOps-shaped testing

Use it when you need fast iteration on manifests, hooks, or application behavior before touching Hetzner-backed environments.

SafeOps Baseline

In the current SafeOps implementation:

  • Terraform manages the lifecycle of the local kind cluster.
  • the cluster is multi-node, so scheduling behavior is closer to reality than a single-node toy setup.
  • Flux Operator + FluxInstance can bootstrap the local cluster from the same GitOps layout.
  • local registry support keeps image iteration fast.

Investigation Snapshots

Here is the Terraform module layout used for the local cluster in the SafeOps system.

Local kind cluster Terraform module

Show the local cluster Terraform layout
  • infra/terraform/kind_cluster/.gitignore
  • infra/terraform/kind_cluster/Makefile
  • infra/terraform/kind_cluster/README.md
  • infra/terraform/kind_cluster/UPGRADE.md
  • infra/terraform/kind_cluster/local-profile.tf
  • infra/terraform/kind_cluster/main.tf
  • infra/terraform/kind_cluster/scripts/merge-kubeconfig.sh
  • infra/terraform/kind_cluster/terraform.tfvars.example
  • infra/terraform/kind_cluster/values/components.yaml
  • infra/terraform/kind_cluster/variables.tf

Here is the local development runbook used in the SafeOps system.

Local development runbook

Show the local development runbook
This repo supports:
- a local `kind` cluster (fast feedback loop), and
- a Hetzner Cloud cluster (provider-realistic).

If you are using Hetzner as the primary environment, start with `docs/hetzner.md`.

Use Terraform to provision a local multi-node kind cluster. Terraform manages lifecycle and kubeconfig generation so you can focus on deploying workloads.

## Local profile (what you get on kind)

`terraform apply` with the defaults reconciles `flux/bootstrap/profiles/local`: the same GitOps tree as production **minus** everything that needs a cloud account, a DNS zone or the platform's SOPS key:

| Included on kind | Excluded on kind (the cloud profiles only) |
|---|---|
| namespaces, NetworkPolicies, ResourceQuotas/LimitRanges | cert-manager, external-dns, Cloudflare DNS/TLS |
| backend + frontend in develop/staging/production | Dex, Headlamp, OIDC RBAC |
| kube-prometheus-stack (Prometheus, Grafana) | Uptrace export of the collectors (on kind they print to their own log) |
| CloudNativePG operator + one cluster per environment | Slack delivery (the Guardian logs its alerts instead) |
| **MinIO** as the CNPG backup target (replaces Hetzner Object Storage) | Flux image automation (needs a git write token) |
| Kyverno + both policy packs (Audit) | `flux/secrets/*` encrypted for the platform key |
| Flagger (develop canaries are opt-in, Ch19), Chaos Monkey | |
| The OpenTelemetry collectors (pod logs, Kubernetes Events; `docs/observability/logs.md`) | |
| **The Guardian** (k8s-ai-monitor, Ch14): Alertmanager hands it every alert but `Watchdog` and `InfoInhibitor`; with your own LLM key (below) it analyses production incidents | |
| `secrets-local`: your own SOPS secrets under `flux/secrets/local/` | |
| `lab` namespace: Pod Security *warn/audit* instead of enforce, for drills that must observe a violation (Kyverno reports) | |

Runtime secrets the apps need (`backend-secrets`, `app-postgres-app`, `cnpg-backup-s3`, `minio-credentials`, `sops-age`, `k8s-ai-monitor-secrets`) are generated by Terraform per cluster and never committed.

**The Guardian's LLM key** (Chapter 14) is yours, set in `infra/terraform/kind_cluster/terraform.tfvars` (git-ignored) and applied with `make kind-plan` / `make kind-apply`:

```hcl
guardian_llm_provider = "gemini"        # gemini (cheapest), anthropic or openai
guardian_llm_api_key  = "your-own-key"

Without a key the Guardian still detects, deduplicates, tracks and posts incidents - only the LLM analysis and the daily report are missing. With a key, every request it sends and the answer are kept for 7 days (GET /llm-debug, turned on by the local profile): you can see exactly what left the cluster. Its HTTP API token (internal-token) is generated; read it with kubectl --context kind-sre-control-plane -n observability get secret k8s-ai-monitor-secrets -o jsonpath='{.data.internal-token}' | base64 -d. The age key lands in age.agekey at the repo root (git-ignored); register its public half with scripts/sops-setup.sh --local before you encrypt anything.

Images are pulled from ghcr.io/safeops-course/*; no registry credentials are needed as long as the packages are public.

Prerequisites

  • Docker Engine with at least 4 CPUs and 8 GB of memory for the two-node kind cluster (12 GB recommended); make check-tools verifies it
  • curl, tar, and unzip available on your workstation
  • Go 1.24+ and Node.js 20+ with npm for backend/frontend development
  • make (GNU make recommended)
  • Terraform 1.11+, kubectl, pre-commit, checkov at least the version CI runs (the Terraform security hook fails without it or with an older one; Homebrew lags behind, so uv tool install checkov==<version> - make check-tools prints it), jq (the AI agent’s kube-context hook, Chapter 01, and scripts/lab-pod.sh) and yq v4 (the pre-commit guardrails that read the Flux manifests) - make check-tools checks that every tool is installed, and its minimum version where one is set

Provision the Cluster with Terraform

Use the Terraform module under infra/terraform/kind_cluster/ to create (or destroy) the local kind cluster. The module codifies the multi-node topology directly in Terraform, automatically merges the generated kubeconfig into ~/.kube/config, and bootstraps Flux via Flux Operator + FluxInstance. GitOps reconciliation is on by default (course repo, local profile, anonymous HTTPS - the repository is public). Working from a fork, put its URL into infra/terraform/kind_cluster/terraform.tfvars (git-ignored; start from terraform.tfvars.example) - not into an export: a plan from a terminal without it would point Flux back to the course repository and reinstall Flux:

cp infra/terraform/kind_cluster/terraform.tfvars.example infra/terraform/kind_cluster/terraform.tfvars
### Name the Cluster in Every Command
Terraform merges the kubeconfig into your default config (`~/.kube/config`) under kind's context name `kind-sre-control-plane`, and also writes it next to the module (`kubeconfig.yaml`). Do not switch the current context: it is one line in a file every terminal shares, and another terminal - or an AI agent - can change it between your check and your command (Chapter 01). Name the cluster instead:
```bash
kubectl --context kind-sre-control-plane get nodes
flux --context kind-sre-control-plane get kustomizations -A

make smoke-test and scripts/lab-pod.sh name it for you (KUBE_CONTEXT, default kind-sre-control-plane).

Run a local container registry to speed up iterative image pushes:

docker run -d --restart=always -p 5001:5000 --name kind-registry registry:2

Ensure the registry is running before applying Terraform so the mirror entry in the cluster configuration is valid.

Destroy the Cluster

Use Terraform to tear down the environment when finished:

cd infra/terraform/kind_cluster
terraform destroy

Test Pods in the Environment Namespaces

develop, staging and production enforce Pod Security restricted; a plain kubectl run busybox is rejected before it starts. Use the wrapper, which adds the required securityContext, waits, prints logs and returns the command’s exit code:

For pods that are *meant* to be non-compliant (the admission-policy audit drills) use the `lab` namespace.

## Verify the Cluster Is Green
```bash
flux --context kind-sre-control-plane get kustomizations -A    # every row Ready=True
kubectl --context kind-sre-control-plane -n develop get pods   # backend, frontend, app-postgres-1 Running
kubectl --context kind-sre-control-plane -n minio get pods     # minio Running, minio-create-bucket Completed
make smoke-test                                               # tests/smoke-test.sh, on kind-sre-control-plane

Pods stuck in ImagePullBackOff mean the GHCR packages are not public for your account - see the note above.

Next Steps

  • Review docs/gitops/flux.md for Flux usage; controllers are installed automatically by Terraform.
  • Build and run the backend locally: cd backend && go run ./cmd/api, then curl http://localhost:8080/healthz or scrape http://localhost:8080/metrics.
  • Build the container image via make -C backend image and push it to your preferred registry (kind can use the local mirror at localhost:5001).
  • Publish the production-ready image to GitHub Container Registry with make backend-publish (or make -C backend publish). Export DOCKER_PAT (PAT with write:packages) and optionally DOCKER_USER beforehand; override REGISTRY_HOST/REGISTRY_NAMESPACE/IMAGE_NAME/TAG as needed.
  • The image build embeds git metadata (APP_VERSION, APP_COMMIT, APP_COMMIT_SHORT, APP_BUILD_DATE) via Go ldflags; APP_VERSION defaults to the latest annotated tag (SemVer). Override it when publishing a release and verify /version reflects your build (including build_time).
  • Run make test (executes Go unit tests; Vue tests pending) before committing.
  • Explore the API via http://localhost:8080/swagger for Swagger UI or http://localhost:8080/openapi for the raw spec.

Safe Workflow (Step-by-Step)

The course path is Chapter 00; this appendix is the reference behind it.

  1. Confirm local prerequisites first: Docker Engine, Terraform, kubectl, make, and at least 4 CPUs / 8 GB of memory for Docker (12 GB recommended) for the two-node kind cluster - make check-tools verifies all of it.
  2. Move to infra/terraform/kind_cluster/. By default Flux reconciles the local profile (flux/bootstrap/profiles/local) from the course repository; point flux_git_repository_url in terraform.tfvars (from terraform.tfvars.example, ignored by Git) at your fork, and TF_VAR_flux_kustomization_path=./flux/bootstrap/profiles/hetzner (the full platform - the Hetzner profile) only if you have the platform’s SOPS key and cloud secrets.
  3. TF_VAR_flux_git_token is only needed if your fork is private; the platform repository itself is public.
  4. Run terraform init and terraform apply from the kind_cluster module.
  5. Do not switch the current context - it is shared by every terminal using the kubeconfig (Chapter 01). Pass --context kind-sre-control-plane to every kubectl and flux command instead, as the commands below do.
  6. Verify nodes, namespaces, and Flux controllers before testing workloads.
  7. If you iterate on images, start the local registry before cluster apply so the mirror config is valid.
  8. Tear the cluster down with terraform destroy when the test cycle is finished.

Verification Commands

cd infra/terraform/kind_cluster
terraform init
terraform apply

export KUBECONFIG="$(pwd)/kubeconfig.yaml"
kubectl --context kind-sre-control-plane get nodes
kubectl --context kind-sre-control-plane -n flux-system get pods

Optional local registry:

docker run -d --restart=always -p 5001:5000 --name kind-registry registry:2

When to Prefer Local Development

Prefer the local path when:

  • you are validating manifests, hooks, or GitOps wiring
  • you need a fast loop for backend or frontend changes
  • you want to reproduce a failure without risking shared environments

Do not treat the local path as a substitute for provider-realistic verification. Hetzner, external DNS, cloud load balancers, and real certificate issuance still need cloud-side validation.

Guardrail Principle

Use the local cluster to shrink the feedback loop and the blast radius. Use the cloud cluster to validate provider-specific behavior. Do not confuse the two.

Done When

  • you can create and destroy the local cluster from Terraform
  • kubectl can target the local context without ambiguity
  • Flux reconciles the local profile and every Kustomization is Ready (flux --context kind-sre-control-plane get kustomizations -A)
  • you can explain which tests belong locally and which still require cloud validation