Production-Grade Kubernetes with Guardrails & AI-Assisted SRE

This page is the detailed study plan. Use /course/ for the course overview and learner-facing homepage. Use this page when you want the exact order, difficulty, time estimate, and prerequisites for every chapter.

Core Track (15 Chapters + Setup)

  1. Your Lab Environment Setup · ~1h
  • fork of the platform repository, local kind cluster reconciled by Flux (local profile)
  • your own SOPS key, the lab-pod.sh wrapper, the lab namespace
  • the three baseline commands every later chapter starts with
  • Prerequisites: Docker, Terraform, kind, kubectl, flux, age/sops; a GitHub account (fork of the platform repository)
  1. Blast Radius & the Shape of Safety Beginner · ~25 min
  • one correct command against the wrong cluster: how far a mistake can spread
  • the four rules for every change: verify context, plan before apply, one failure domain, evidence before merge and after deploy
  • the layers that stop a mistake, and which part of the course builds each one
  • Prerequisites: Basic Kubernetes familiarity (pods, deployments, services)
  1. Infrastructure as Code with Terraform Beginner · ~1.5h
  • Terraform’s state as its memory, and how a plan is made from the code, the state and the real objects
  • one memory for the team: a shared backend with a lock (Cloudflare R2 for Hetzner, MinIO in the lab)
  • apply only the plan you have read, while it is fresh: the guard on a workstation, a hash and an age limit in CI
  • drift with plan -detailed-exitcode: revert, codify or stop; secrets in plans and state; what a cloud destroy leaves behind
  • lab on the local kind cluster: two local states create the same objects twice, a shared locked state fixes it, and a stale plan undoes a hand fix
  • Prerequisites: Chapters 00-01; basic CLI/terminal skills
  1. GitOps with Flux Intermediate · ~1.5h
  • how Flux turns Git into cluster state: sources, Kustomizations, reconciliation
  • prune, drift correction, and why a hand change does not survive the next reconcile
  • rollback with git revert, and the one place where Flux decrypts secrets
  • Prerequisites: Chapter 02; Git branching familiarity
  1. Secrets Management (SOPS) Intermediate · ~1.5h
  • encrypted before it leaves your machine: SOPS + age, and Flux decrypting in the cluster
  • where a value is still plaintext, and the check that stops a plaintext Secret before the commit
  • leak response: a pushed secret is burned - replace it at its source; age key rotation with updatekeys + rotate
  • Prerequisites: Chapters 02-03
  1. CI/CD & Developer Guardrails Intermediate · ~1.5h
  • every check twice: Git hooks on your machine, the same hooks again in CI - and what --no-verify skips
  • required versus advisory checks: a cancelled check is not a passed one
  • Terraform: a merge only plans; applying is a deliberate, approved run of a fresh plan
  • Prerequisites: Chapters 02-04; Git branches and pull requests
  1. Network Policies (Production Isolation) Intermediate · ~1.5h
  • why a flat cluster network makes every pod as dangerous as the most exposed one
  • default deny, one written-down rule per path - and why policies only add
  • proving paths with a probe that separates DNS from TCP; the few internet rules and their remaining risk
  • Prerequisites: Chapters 01, 03 and 05; basic TCP/IP networking
  1. Security Context & Pod Hardening Intermediate · ~1.5h
  • what each setting takes away from a compromised process - non-root, read-only image, no capabilities, seccomp, no API token - and what it does not
  • three layers: the check in the pull request, Pod Security at admission, the running container
  • “Read-only file system” vs “Permission denied”: a size-limited volume, never root
  • Prerequisites: Chapters 01, 05 and 06; Linux users and file permissions
  1. Resource Management & QoS Intermediate · ~1.5h
  • requests and limits for the scheduler, the quota and the kernel: CPU is throttled, memory is killed
  • who stopped the container - OOMKilled, eviction, throttling, the quota, the LimitRange - and where each leaves its trace
  • QoS classes decide eviction order, not priority; the platform’s budget and the worst-case check before a merge
  • Prerequisites: Chapters 03, 05 and 07; CPU and memory basics
  1. Availability Engineering (HPA + PDB) Intermediate · ~1.5h
  • one owner for the replica count: the HPA, not Git - and the check that keeps it so
  • what a PodDisruptionBudget guards (evictions) and what it does not (rollouts, crashes); minAvailable vs maxUnavailable
  • a drain prepared with evidence - placement, room for replacements, every budget - and simulated with --dry-run=server
  • Prerequisites: Chapters 03 and 08
  1. Version Promotion Intermediate · ~1.5h
  • develop and staging are built, production is promoted: a re-tag of the tested staging image, never a rebuild
  • identity proven by digest - in the registry, in the running pods - not by a tag’s name
  • the ImagePolicy that picks production’s version, and a rollback that pins it instead of fighting image automation
  • Prerequisites: Chapters 03, 05 and 09
  1. Observability (Metrics, Logs, Traces) Intermediate · ~1.5h
  • alerts that ask for an action: silence at rest, a heartbeat, user requests per environment as the signal
  • error-budget burn rates with two windows and a traffic floor; routing, grouping and muting by environment
  • optional, with your own Uptrace account: one request traced from the browser through the backend to the database, and to its log line by trace id
  • Prerequisites: Chapters 08-10; familiarity with Prometheus concepts
  1. Backup & Restore Intermediate · ~2.5h
  • why PostgreSQL and CloudNativePG; base backups and the WAL archive
  • a point-in-time restore into a new cluster, proven by the data - not by Ready
  • recoverability alerts, and a measured restore time against the RTO
  • Prerequisites: Chapters 06, 08 and 11; PostgreSQL basics
  1. Controlled Chaos Intermediate · ~3h
  • chaos as an experiment: steady state, a written hypothesis, one fault, a measurement
  • an outage measured through the Service, not with kubectl get pods
  • a Chaos Monkey the cluster keeps on a leash - RBAC, an admission policy - and a full stop
  • Prerequisites: Chapters 09, 11 and 12
  1. AI-Assisted SRE Guardian Intermediate · ~2.5h
  • Alertmanager, events, Flux and scanners into one incident per signal with k8s-ai-monitor
  • what reaches the LLM (production only, redacted, capped) and what never does
  • an analysis read as a hypothesis; limits the cluster enforces: read-only, no Secrets, an API token
  • Prerequisites: Chapters 11 and 13
  1. 24/7 Production SRE Intermediate · ~3h
  • one incident run end to end: severity from the impact, roles, escalation and handover
  • enough evidence for each decision; recovery proven on the users’ path, not by an alert
  • a blameless postmortem whose actions have an owner, a date and a check
  • Prerequisites: Chapters 10, 11 and 14 (and the habits of 01-14)

Advanced Track

  1. Admission Policy Guardrails (Advanced) Advanced · ~1.5h
  • Kyverno at admission: Audit records, Enforce refuses - per rule and per namespace
  • Enforce where the PolicyReports say it is safe; the image rules read the reference the runtime pulls
  • exceptions that are narrow, owned, temporary and only from Git; a policy engine that fails closed when it crashes - and whose admission checks stop when its admission controller is scaled to zero
  • Prerequisites: Core track complete
  1. Supply Chain Security (Advanced) Advanced · ~1.5h
  • keyless signing in CI: the certificate names workflow, repository and branch; SBOM and provenance with it
  • verified at admission by the cluster’s own verifier - and every signing change tested against real images
  • a narrow signer identity and fail closed in the application namespaces; the verified digest pinned into the pod closes the tag-movement gap (Kyverno 1.19), and an image the mutation did not pin is refused
  • Prerequisites: Chapter 16
  1. Rollback and Data Migrations (Advanced) Advanced · ~1.5h
  • an image rollback does not roll back a schema; a point-in-time restore takes back the users’ writes - neither undoes a release
  • expand/contract and roll forward: the previous release always runs on the new schema, checked in CI before a migration merges
  • migrations in an initContainer from the same image; feature flags that separate writing from reading; a restore only into a separate cluster
  • Prerequisites: Chapters 10 and 12
  1. Progressive Delivery (Advanced) Advanced · ~1.5h
  • a canary limits how many requests a bad release gets, and for how long: a share of the requests first, its own success rate and latency decide
  • a check with no data is a failure: metrics of the canary’s requests and traffic through the split, an alert when a release is rolled back
  • the frontend in develop with Flagger and Traefik; why a service with data is out of scope
  • Prerequisites: Chapters 11, 15 and 18
  1. Platform Lifecycle (Advanced) Advanced · ~1h
  • nothing on the platform moves by itself: pinned charts, database image, Kubernetes and OS on the nodes; everything moves through a reviewed pull request
  • a monthly dependency dashboard instead of a pull request per update; release notes read, small steps, tests that run the cluster’s versions
  • node maintenance at a chosen time, and the honest cost of one worker with every database instance on it
  • Prerequisites: Chapters 09 and 17
  1. Credential & Access Lifecycle (Advanced) Advanced · ~1.5h
  • a registry of every credential: what it opens, where every copy lives, when it is due - checked every morning
  • a rotation ends with two proofs: the new value works, the old one is refused; readers restarted through Git
  • people get access by team with least privilege, short token lifetimes decide how fast it ends; the admin kubeconfig is break-glass
  • Prerequisites: Chapters 04 and 20
  1. Disaster Recovery (Advanced) Advanced · ~1.5h
  • for each failure - a pod, a disk, a node, the cluster, the provider - what brings it back, its RTO and RPO, measured
  • production’s database as a primary and an asynchronous streaming replica; a switchover and a crash timed from the application’s side
  • one backup archive per cluster - a rebuilt cluster must not write into its predecessor’s; the cluster lost and rebuilt, the data restored through Git
  • Prerequisites: Chapters 12, 20 and 21
  1. Capstone: Prove and Hand Over Your Platform (Advanced) Advanced · ~3-4h
  • one change - display names in staging, then production - with a goal, constraints and acceptance criteria, and no commands to follow
  • a colleague’s merged change that passes every check and still breaks the platform: noticed, proven from its symptoms, fixed through Git
  • an AI agent’s plausible shortcut accepted or rejected with a reason; a handover where every claim has its evidence; make capstone-check for what a script can prove
  • Prerequisites: Chapters 16-22

Reference Appendices

  1. Local Development Environment Reference · ~45 min
  • Terraform-managed kind cluster for fast validation
  • local kubeconfig, optional local registry, and local Flux bootstrap
  • when to use local verification versus provider-realistic cloud validation
  • Prerequisites: Chapter 02 recommended
  1. DNS and TLS Automation Reference · ~45 min
  • external-dns + cert-manager as the SafeOps edge automation baseline
  • Cloudflare proxied DNS, Cloudflare Origin CA (ClusterOriginIssuer), Full (strict), and Traefik ingress TLS wiring
  • DNS and certificate failures as edge incidents, not application incidents
  • Prerequisites: Chapters 04 and 11 recommended