The Incident: The Upgrade Put Off Since the First Commit

Incident Hook

This one happened on this platform.

The platform’s admission policies run on Kyverno, installed from a Helm chart pinned in Git: chart 3.7.1, Kyverno 1.17. The version was set in the repository’s first commit and never touched again. Pinning was right - nothing moved by itself. But nothing moved at all.

Meanwhile the policies grew. Their tests moved to the Kyverno CLI 1.19: 1.17’s CLI counted a wrong expectation as passed when the test loaded exceptions - a test that could not fail. So the tests ran 1.19, and the cluster ran 1.17. Every test was green, and said nothing about the engine that decided admission. Chapter 17’s supply-chain policy then needed a feature - writing the verified digest into the pod - that 1.17 does not implement for its policy type. Set to true, it silently did nothing; that was found by running the lab on kind, not by any test.

So the upgrade came, two minor versions at once, in one pull request:

3.7.1 -> 3.9.1    Kyverno 1.17 -> 1.19    one pull request, tests green

What happened on kind, in order:

  1. The upgrade timed out. Helm waits for every Kyverno pod to be Ready - by default 5 minutes. On a slow network each new image took about two minutes to pull. At 5 minutes Helm called the upgrade failed, and Flux rolled it back to 3.7.1. The retry went through: the images were already on the node.
  2. The new version did something the old one did not. The policy had asked, since its first version, for its rules to be generated for Deployments, Jobs and the other pod controllers as well (“autogen”). On 1.17 that had no effect. On 1.19 it did: every Flux dry run of a Deployment now verified its images again, ran past the 30-second webhook timeout on the slow network, and failed - and the policy fails closed. backend and frontend could not be applied in develop, staging and production.
  3. A copy that should only report, refused. MinIO’s bucket Job was denied by the audit copy of the policy - the one that never denies: the generated rule for Jobs expected an annotation that only the pod mutation wrote. That check was 1.17’s - 1.19 had dropped it before its release - so this refusal came from the old version, while it still answered.
Deployment/develop/backend dry-run failed: failed calling webhook "ivpol.validate.kyverno.svc-fail-finegrained-verify-images-enforce": ... context deadline exceeded
Job/minio/minio-create-bucket dry-run failed: ... denied the request: kyverno.io/image-verification-outcomes annotation not present

Running pods were not touched. But nothing in those environments could be updated until the next pull request switched the generated rules off.

Result: a platform that could not deploy until the next pull request, caused by a version jump that had been waiting since the first commit - and found on kind only because the change ran there before anywhere else.

Observed Symptoms

  • The test suite green on every pull request, while the cluster ran a different version of the engine the tests exercised.
  • A Helm upgrade that failed and rolled itself back, with nothing broken in the chart: timeout waiting for: [Deployment/kyverno/kyverno-admission-controller status: 'InProgress' ...].
  • Around the upgrade: Kustomizations not Ready with webhook timeouts on Deployment dry runs; a Job refused by a policy copy that only audits.
  • A dependency dashboard, the first time it ran: about twenty updates waiting - Kubernetes one minor behind, the monitoring stack ten major versions behind, the database operator two releases behind.

Confusion Phase

The first explanations pointed everywhere but the cause:

  • “The chart is broken - roll back and stay on 3.7.1.”
  • “The network is slow, it will pass by itself.”
  • “The tests pass, so the policy is fine; something else changed.”

Each had a grain of truth - the network was slow, the tests did pass - and each would have left the platform where it was: on a version two releases behind, with tests that check another one.

What AI Would Propose (Brave Junior)

  • “Turn on automatic updates for everything, then it can never fall behind again.”
  • “Raise the webhook timeout and the Helm timeout to an hour.”
  • “Unpin the chart - version: 3.x always gets the latest fixes.”
  • “Skip the minor versions in between, one jump is less work.”

Why This Is Dangerous

  • A version that moves by itself moves when you are not looking. Unpinned charts and automatic node updates trade a known old version for an unknown new one, on a day nobody chose.
  • Not updating is not standing still. Every month without an update adds changes the next update has to absorb at once - the 1.17 -> 1.19 jump brought a timeout, a behaviour change and a refusal in one pull request.
  • Tests against another version prove nothing about yours. The policy tests ran 1.19 and passed while the cluster ran 1.17. A check of what runs is part of the test, not an extra.
  • Longer timeouts hide, they do not fix. The Helm timeout was right to be raised - pulling images takes the time it takes. The webhook timeout was not the problem: verifying Deployments at all was.
  • Skipping versions skips the warnings. Release notes of the versions in between are where removals and changed defaults are announced.

Pause and Predict: The tests of a policy pass on every pull request. The cluster runs an older version of the policy engine than the tests. What does a green test tell you about admission on the cluster - and what one check would close the gap?