Lab & Completion

What You Will Produce

On your own kind cluster, the guardrails at work. You read each guardrail’s two policies - one that refuses, one that records - and the PolicyReports per namespace, and see why the application namespaces enforce while the platform namespaces only record. You run the same untagged pod twice: admitted with a report entry in lab, refused with the reason in develop - then fixed with a tag. You take one rule from Audit to Enforce in lab the way it is done: evidence first, the workload fixed, then a pull request - and see the untagged pod refused there too. You weaken a rule on a branch and watch its test refuse the change. You give one named pod an exception through a pull request, see a second pod still refused, and see the same exception refused when applied by hand. Then you switch the policy engine’s admission checks off for a minute - for the whole cluster - to see what a stopped Kyverno leaves open, and why Flux keeps working. At the end, the Enforce change and the exception leave the way they came.

Before You Start

The baseline from Chapter 00 must be clean:

flux --context kind-sre-control-plane get kustomizations -A          # every row READY=True and SUSPENDED=False
kubectl --context kind-sre-control-plane get pods -A --no-headers | grep -vE 'Running|Completed'   # prints nothing
make smoke-test                     # 9/9

You need curl, jq, gh signed in to GitHub, and the Kyverno CLI 1.19 or newer - new in this chapter; make check-tools lists it. On macOS brew install kyverno; on Linux the kyverno-cli_v<version>_linux_x86_64.tar.gz archive from https://github.com/kyverno/kyverno/releases, with kyverno moved onto your PATH. Check: kyverno version. Your working tree must be clean (git status --short prints nothing): B6 restores a policy file. Your fork must be up to date: if kubectl --context kind-sre-control-plane get namespace policy-exceptions says NotFound, open your fork on GitHub, click Sync fork → Update branch, then git switch main && git pull, and let Flux catch up.

The Lab in Three Parts

  • A. Verify - the baseline, each rule’s mode, and what the PolicyReports say per namespace.
  • B. Break & Fix - one pod admitted in Audit and refused in Enforce, then fixed; one rule taken to Enforce in lab on evidence; a weakened rule caught by its test; an exception through a pull request and one refused by hand; the policy engine switched off.
  • C. Rollback & Cleanup - the Enforce change and the exception reverted through pull requests, the lab pods deleted, the baseline.

Open the lab →

Done When

  • you can say which policies enforce where, and point at the namespaces whose reports keep the rest in Audit (A2-A3)
  • your prediction was written before the first pod (B1)
  • the same untagged pod was admitted in lab with a fail in its report, and refused in develop with the rule’s message (B2-B3)
  • a tag fixed it, and you can say why “no tag” counts as latest (B4)
  • lab had no disallow-latest-tag failures left after you fixed the workload, the rule was enforced there through a pull request, and the untagged pod was refused in lab (B5)
  • a weakened rule failed tests/kyverno-policies.test.sh, and the branch was thrown away (B6)
  • an exception for one named pod went in through a pull request, admitted that pod and no other, and the same exception applied by hand was refused (B7-B8)
  • with Kyverno’s admission controller scaled to zero, its webhooks were gone and an untagged pod was admitted in develop; Flux - outside the webhooks - brought the engine back from Git, the webhooks were registered again and the same pod was refused again (B9)
  • the Enforce change and the exception left through pull requests, no lab16- pod is left, and the baseline is clean (C1-C3)

Knowledge Check

Before finishing this chapter, complete the Quiz.