Chapter 05: CI/CD & Developer Guardrails

Why This Chapter Exists

CI/CD pipelines are where code becomes infrastructure. Without guardrails at every stage, a single unvalidated change can bypass all cluster-side protections. This chapter defines the layered defense model: local hooks, CI validation, approval gates, and AI-assisted review.

Incident Hook

A developer pushes directly to main, skipping validation. Terraform applies an untested change. An unsigned image reaches production. Each failure happened because one guardrail layer was missing or bypassed.

What AI Would Propose (Brave Junior)

• “Skip pre-commit hooks locally, CI will catch it.”
• “Apply Terraform directly, we already know what it does.”
• “Merge without review, the change is small.”

Why this sounds reasonable:

• faster iteration in the moment
• fewer steps to production
• small changes feel safe

Why This Is Dangerous

• CI cannot catch what was never pushed (secret committed and force-pushed away still leaks).
• Direct apply without plan review removes the last safe checkpoint.
• Small unreviewed changes accumulate into unauditable drift.

Guardrails That Stop It

• Pre-commit hooks block risky operations before code leaves the workstation.
• CI pipelines enforce validation, scanning, and approval gates.
• AI-assisted review catches patterns humans miss under pressure.
• Admission policies (Chapter 16) provide the final cluster-side gate.

Core Concepts

1. Pre-commit Hooks (First Guardrail Layer)

Pre-commit hooks run before code leaves the developer workstation. They are the cheapest guardrail: fast feedback, zero CI cost, immediate correction.

Branch protection:

• master-branch-check.sh blocks direct commits to main/master. All changes must go through feature branches and PRs.

History safety:

• prevent-amend-after-push.sh blocks amending commits that have already been pushed. This prevents rewriting shared history.

Secret blocking:

• block-secrets.sh pattern-matches on dangerous file types: kubeconfig, .key, .pem, .env, and credential patterns. Catches secrets before they enter Git history.

Flux manifest validation:

• flux-kustomize-validate.sh runs a 3-stage validation pipeline:
  1. YAML syntax check
  2. kustomize build to verify overlay resolution
  3. kubeconform with CRD schemas for structural validation

Terraform validation:

• terraform fmt check ensures consistent formatting
• terraform validate catches configuration errors
• checkov security scan flags misconfigurations

External linters:

• shellcheck for shell script quality
• yamllint for YAML formatting consistency

2. GitHub Actions Pipeline Design

The CI pipeline enforces what local hooks cannot guarantee (because developers can skip hooks).

Plan-Approve-Apply pattern (from terraform-hcloud.yml):

Plan Job → Upload Artifact → Approval Gate → Download Artifact → Apply Job

Key design decisions:

• Concurrency control: cancel-in-progress: false for apply jobs. Never cancel a running infrastructure mutation.
• Artifact passing: tfplan uploaded with 1-day retention. Apply job downloads the exact reviewed plan.
• Environment protection: GitHub environment with required reviewers and 60-minute timeout.
• Secret management: Infrastructure credentials passed via TF_VAR_* environment variables from GitHub Secrets.

Destroy workflow (from terraform-hcloud-destroy.yml):

Destruction requires elevated confirmation:

• Manual trigger (workflow_dispatch) with confirmation string input (“DESTROY”)
• Multi-approver requirement
• Makefile-delegated destroy sequence: Flux/K8s cleanup before terraform destroy

3. AI-Assisted Code Review (CodeRabbit)

CodeRabbit provides automated review as a safety net, not a replacement for human review.

Configuration highlights:

• Path-specific rules: different review depth for infrastructure vs application code
• Profile: “chill” — non-aggressive tone, focuses on real issues
• Security tools integration: gitleaks (secrets), semgrep (code patterns), checkov (IaC), hadolint (Dockerfiles), yamllint (YAML), actionlint (GitHub Actions)
• KISS principle enforcement: flags unnecessary complexity

4. Guardrails Layering Model

Local (pre-commit) → CI (GitHub Actions) → Review (CodeRabbit) → Cluster (Kyverno admission)

Each layer catches what the previous layer missed:

• Pre-commit catches developer mistakes immediately
• CI catches bypassed hooks and validates against real schemas
• CodeRabbit catches patterns and anti-patterns across the full PR
• Admission policies enforce cluster-side invariants regardless of pipeline

No single layer is sufficient alone. Defense in depth means every layer assumes the previous layer failed.

Safe Workflow (Step-by-Step)

1. Install pre-commit hooks: pre-commit install --install-hooks
2. Commit triggers local validation (branch check, secret scan, manifest validation)
3. Push triggers CI pipeline (plan, validate, security scan)
4. PR creation triggers CodeRabbit review
5. Merge to main triggers apply with approval gate
6. Flux reconciles to cluster with admission policies as last gate

Repo Mapping

Platform repository references:

• .pre-commit-config.yaml — hook definitions
• scripts/pre-commit-master-check.sh — branch protection
• scripts/prevent-amend-after-push.sh — history safety
• scripts/block-secrets.sh — secret blocking
• scripts/flux-kustomize-validate.sh — Flux validation
• .github/workflows/terraform-hcloud.yml — CI/CD pipeline
• .github/workflows/terraform-hcloud-destroy.yml — destroy workflow
• .coderabbit.yml — AI review config

Anti-Patterns to Avoid

• Relying on CI alone without local hooks (slow feedback, wasted CI minutes).
• Using --no-verify to skip hooks during “quick fixes.”
• Running apply without approval gate, even for “trivial” changes.
• Ignoring CodeRabbit findings because “it’s just AI.”
• Treating any single layer as the complete guardrail.

Lab Files

• lab.md
• quiz.md

Done When

• learner can explain the guardrails layering model and why each layer exists
• learner can install and trigger pre-commit hooks locally
• learner can trace the Plan-Approve-Apply pipeline flow
• learner can describe how CodeRabbit integrates with the review process
• learner can identify what each guardrail layer catches that others miss

Next Chapter

Continue with Chapter 06 (Network Policies).