SafeOps Academy

Lab: Guardrails Before Speed

Goal

Run the chapter-01 safe path end-to-end:

  • verify Kubernetes context and namespace guard
  • verify Terraform plan-before-apply guard
  • demonstrate blocked unsafe action
  • execute rollback checklist simulation

Prerequisites

  • repository cloned locally
  • pre-commit hooks installed
  • kubeconfig configured for your target cluster/context
make install-hooks
pre-commit run --all-files

Step 1: Validate Kubernetes Context Guard

Successful example:

scripts/guard-kube-context.sh --context sre-control-plane --namespace develop

Expected:

  • explicit OK output for context+namespace

Failure example:

scripts/guard-kube-context.sh --context sre-control-plane --namespace does-not-exist

Expected:

  • command exits non-zero
  • clear namespace mismatch error

Step 2: Validate Terraform Plan Guard

Create plan artifact:

scripts/guard-terraform-plan.sh plan --dir infra/terraform/hcloud_cluster --out tfplan

Then apply from reviewed plan:

scripts/guard-terraform-plan.sh apply --dir infra/terraform/hcloud_cluster --out tfplan --max-age-minutes 60

Step 3: Block an Unsafe Apply

Delete plan metadata (or use stale plan) and retry apply:

rm -f infra/terraform/hcloud_cluster/tfplan.meta
scripts/guard-terraform-plan.sh apply --dir infra/terraform/hcloud_cluster --out tfplan --max-age-minutes 60

Expected:

  • apply blocked with explicit policy error

Step 4: Split Change Types (Simulation)

Prepare two separate change sets:

  • change set A: image/promotion only
  • change set B: ingress/networking only

Validation target:

  • no mixed app+network change in one commit/PR

Step 5: Rollback Drill (Command Rehearsal)

Runbook rehearsal commands:

kubectl -n develop rollout undo deployment/backend
kubectl -n develop rollout undo deployment/frontend

GitOps rollback rehearsal:

  • identify exact commit for ingress/network change
  • prepare git revert <sha>

Evidence to Capture

  • guard success output (guard-kube-context)
  • guard failure output (guard-kube-context or plan apply block)
  • plan/apply command transcripts
  • rollback rehearsal command output

Hard Stop Conditions

  • applying infrastructure without reviewed plan artifact
  • skipping context/namespace verification
  • mixing app and networking changes into one deployment action

Done When

  • learner demonstrates both allowed and blocked paths
  • learner can explain why blocked path is unsafe
  • learner can execute rollback checklist without improvisation