The Incident: The Report Nobody Read

Incident Hook

Monday, 16:20. Production logs are hard to search, and a teammate adds a log-shipping sidecar to the backend: one container, image logship/agent from Docker Hub, no tag. The pull request is green - no check reads the images - and the reviewer approves “just a sidecar”. Flux applies it.

16:21. Kyverno has been installed for months, every policy in Audit. Two new entries appear in the production PolicyReport: an image without a tag, from a registry nobody approved. Nobody reads PolicyReports.

Three weeks later, Thursday 03:10. A node is replaced; the backend pods are scheduled again and pull logship/agent - which, with no tag, means latest. (An image is pulled when a container starts; for three weeks no new backend pod had started, so nothing pulled - with no tag Kubernetes pulls on every start, and latest had moved.) The vendor published a new major version the evening before. The new sidecar exits at start; every backend pod fails its readiness check. Users get errors. Nobody can pin the old version: nobody knows which one it was.

Friday, 10:00. A pull request switches every Kyverno policy to Enforce, in every namespace - “so it never happens again”. It is merged before lunch.

11:40. Flux upgrades the monitoring stack, as it does every week. Its new pods are refused: the charts’ images break the registry rule, some containers have no limits. The old pods are already gone. Monitoring is blind.

12:15. To get monitoring back, someone deletes Kyverno’s webhook configuration by hand. The pods come back. So does every image, from every registry, in every namespace.

Result: a guardrail that only reported protected nothing for three weeks; the same guardrail enforced without evidence broke the platform in an hour; and the way out switched it off for everyone.

Diagram: sidecar, no tag, unknown registry; Audit: report entry nobody reads; three weeks later: 'latest' changes, production fails; Enforce everywhere, no evidence; platform pods refused; webhook deleted by hand: no guardrail

The story is invented. The untagged image that a policy in Audit lets through, the same pod refused where the rule enforces, the platform namespaces that would break under Enforce today, and the exception that can come only from Git are what the lab shows on your cluster.

Observed Symptoms

  • The violation was recorded the minute it happened - in a report with no reader.
  • “Audit” had no end date and no owner: nobody was going to promote it.
  • The first Enforce covered every namespace at once, with no look at what was already breaking the rules.
  • When the rule got in the way, the only way around it was to switch it off - for everything.

Confusion Phase

The team argues about the wrong things:

  • should the reviewer have spotted the missing tag?
  • is Kyverno too dangerous to enforce?
  • should every chart be fixed before any rule can enforce?

Was the problem a policy engine that is too strict - or rules that were never promoted, then promoted blind?

What AI Would Propose (Brave Junior)

  • “Switch every policy to Enforce - Audit obviously does not work.”
  • “Monitoring is down: delete the webhook now, put it back later.”
  • “Add an exception for the whole observability namespace, and for production while we are at it.”

Why This Is Dangerous

  • Audit is a stage, not a guardrail. It records; it refuses nothing. A rule in Audit without a reader and a date for Enforce is a dashboard nobody opens.
  • Enforce without evidence breaks what already runs. The PolicyReports said, for weeks, exactly which namespaces would break. Enforce goes where they are clean - the application namespaces here - and waits where they are not.
  • Deleting the webhook removes the rule for everyone. The exception for one workload is a narrow, owned, temporary object, reviewed like the rule. A switch that turns the rule off globally is not an exception; it is the incident’s next chapter.
  • An exception for a whole namespace is the rule switched off with extra steps. It names one rule, one namespace, the resources by name - and when it ends.
  • The policy engine is a dependency of every deployment. When it is down, the cluster refuses what it covers. That has to be designed - which namespaces it covers, and how many replicas answer - not discovered at 11:40.

Pause and Predict: A policy in Audit says fail for a pod. Is that pod running? And if the same policy were in Enforce, at which moment would you have learned about the problem?