Investigation & Containment

Investigation

Four questions, in this order: what does each rule do where, what do the reports say, what would Enforce refuse, and what happens when the engine itself is down. Every command names the cluster with --context (Chapter 01).

  1. What does each guardrail do, and where? Each is two ValidatingPolicies - one that refuses, one that records - split by namespace:

    kubectl --context kind-sre-control-plane get validatingpolicies -o json | jq -r '.items[] | .spec as $s
  | "\(.metadata.name): \($s.validationActions | join(",")) - namespaces \($s.matchConstraints.namespaceSelector.matchExpressions[0] | "\(.operator) \(.values | join(","))")"'

    Each of the five guardrails prints two lines: <name>-enforce: Deny - namespaces In develop,staging,production and <name>-audit: Audit - namespaces NotIn develop,staging,production. (The supply-chain policy of Chapter 17 is another kind of policy and does not appear here.) A guardrail with only an Audit copy, matching every namespace, only records - the state of the story.

  2. What do the reports say? Kyverno writes a PolicyReport per namespace: what it checked, and what failed. Count the failures per namespace and policy:

    kubectl --context kind-sre-control-plane get policyreports -A -o json | jq -r '.items[] | .metadata.namespace as $ns
  | .results[]? | select(.result == "fail") | "\($ns) \(.policy)"' | sort | uniq -c | sort -rn

    On this platform the application namespaces have no failures of the five guardrails; the platform namespaces have many - their charts run images from other registries and leave limits out. In the story the production report had two failures for three weeks. A report is evidence only when someone reads it - or when a rule acts on it.

  3. What would Enforce refuse? Exactly what the report lists as fail - and what it cannot list: workloads that do not exist yet, or not right now. A lab pod, a Job that runs once a night, a restore that creates pods only during an incident (Chapter 12). Before a rule enforces in a namespace, both must be clean: the report, and the occasional workloads, checked by hand. Here, the lab pods used curlimages/curl without a tag; they had to be pinned before the tag rule could enforce in develop.

  4. What happens when the engine is down? It depends on how it went down. Kyverno answers the API server through webhooks with failurePolicy: Fail: when Kyverno crashes - a pod killed, a node lost - and no replica answers, every request the webhooks cover is refused. When Kyverno’s admission controller is stopped on purpose - scaled to zero - it deletes its own webhook configurations on the way out: then no Kyverno rule is checked at admission, in any namespace, until it starts again. Its background scans keep reporting, and the native ValidatingAdmissionPolicies keep refusing; uninstalling Kyverno removes the scans too. Which namespaces do the webhooks cover?

    kubectl --context kind-sre-control-plane get validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg -o json | jq -r '.webhooks[]
  | "\(.name): failurePolicy=\(.failurePolicy), namespaces: \([.namespaceSelector.matchExpressions[] | "\(.operator) \(.values | join(","))"] | join("; "))"'

    Kyverno registers one webhook per kind of policy and failure policy: the enforce copies’ (vpol.validate...-fail-..., failurePolicy=Fail, In develop,staging,production) and the audit copies’ (...-ignore-..., Ignore, NotIn develop,staging,production) - and every one also says NotIn kube-system,flux-system; NotIn kyverno. So every namespace except kube-system, flux-system and kyverno. While Kyverno does not answer, the Fail webhook refuses in the application namespaces; the Ignore one lets the platform namespaces through unrecorded. Two replicas, one per node, and a PodDisruptionBudget keep one answering through a node drain; flux-system is left out so that Flux - the way a broken Kyverno is repaired - never waits for Kyverno. And stopping Kyverno’s admission controller switches every Kyverno admission check off, so who may do it matters as much as the rules. Directly in the cluster, only the break-glass kubeconfig can - the everyday OIDC roles have no write access in the kyverno namespace. Through Git, a reviewed pull request that scales or removes Kyverno’s HelmRelease would have Flux do it - which is why changes to flux/infrastructure/policy/ deserve the strictest review.

The root cause of the story is not a strict engine. It is a rule never promoted, then promoted blind: Audit with no reader and no end, Enforce with no evidence and no scope, and no exception path - so the only way around the rule was to remove it.

Containment

  1. Fix the workload, not the rule. The untagged sidecar gets a tag - the one that worked, from the registry’s history or the old pod’s imageID - through a pull request.
  2. Enforce where the evidence is clean. Only the namespaces whose reports and occasional workloads pass; every other namespace stays in Audit, in the same pull request, with the evidence in its description.
  3. Let one workload through on purpose - narrowly. A workload that cannot be fixed today gets an exception: one policy, one namespace, the workload by name, an owner, a reason and an expiry - through a pull request, like the policy.
  4. Never switch the rule off for everyone. Deleting the webhook, or setting a policy to Audit to get one pod in, removes the guardrail for every workload. If the engine itself is broken, repair it through Flux - which is why flux-system is outside the webhooks.

A guardrail is a rule that refuses where the evidence says it is safe, records everywhere else, and has one narrow, reviewed way through.

Pause and Predict: You promote the tag rule to Enforce in develop. Its PolicyReport is clean. What could still be refused there tomorrow - and how would you have found it today?