Investigation & Containment

Investigation

Five questions, in this order: what runs, what the tests run, what the upgrade changes, whether the upgrade finished, and what else moves by itself. Every kubectl and flux names the cluster with --context (Chapter 01).

What runs? The version Git pins is what Flux asks for; the version the cluster runs is what it got:
```
flux --context kind-sre-control-plane get helmreleases -A
kubectl --context kind-sre-control-plane -n kyverno get deploy kyverno-admission-controller \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
```
The chart version per release, and the image the chart put there: reg.kyverno.io/kyverno/kyverno:v1.19.1. A HelmRelease whose upgrade failed and was rolled back still shows the old revision here - the message says so.
What do the tests run? The policy tests use the Kyverno CLI, the alert tests promtool - both pinned in the CI workflow. tests/tool-versions.test.sh asks Helm what the pinned charts install and compares:
```
tests/tool-versions.test.sh
```
ok Kyverno ...: tests 1.19.1 = chart in Git 1.19.1. In the story the same check would have printed the tests run 1.19.1, the chart in Git installs 1.17.1 - on the first pull request after the tests moved, not months later.
What does the upgrade change? Not what the version number suggests - what the chart renders and what the release notes say. Render both versions and compare:
```
repo=https://kyverno.github.io/kyverno/
diff <(helm template kyverno kyverno --repo "$repo" --version 3.7.1 -n kyverno | grep -E '^\s+image: ' | sort -u) \
     <(helm template kyverno kyverno --repo "$repo" --version 3.9.1 -n kyverno | grep -E '^\s+image: ' | sort -u)
```
Every component from v1.17.1 to v1.19.1, and one surprise: a readiness-checker:latest image - only in Helm test pods, which Flux does not run, so harmless here; found only because the diff was read. The behaviour changes - such as generated rules for pod controllers starting to act - are not in a diff of images; they are in the release notes of every version in between.
Did the upgrade finish? A HelmRelease remembers each attempt:
```
kubectl --context kind-sre-control-plane -n flux-system get helmrelease kyverno \
  -o jsonpath='{range .status.history[*]}{.chartVersion} {.status}{"\n"}{end}'
```
Newest first. In the story, between the timeout and the retry: 3.7.1 deployed - the rollback - over 3.9.1 failed. After the retry went through, the history keeps only the latest releases: 3.9.1 deployed, 3.7.1 superseded - the failed attempt is gone from it, so look while it happens, or in Flux’s events. A rollback puts the release back - the chart’s templates and values - not necessarily the whole platform: CRDs the chart applied (the monitoring stack’s use CreateReplace), what its hooks did and changes in data can stay. After a rollback, check that the platform works, then ask why it timed out.
What else moves by itself? Anything not pinned, and anything set to update itself:
```
grep -nE 'auto_upgrade|kured_enabled|k3s_version' infra/terraform/hcloud_cluster/terraform.tfvars
grep -rn 'imageName:' flux/infrastructure/data/cnpg-clusters/
```
On this platform: nothing on the nodes (auto_upgrade_k3s, auto_upgrade_os, kured_enabled all false), the database image pinned to a minor and its digest. Before this chapter the nodes installed OS updates and rebooted on their own, and the database image followed the moving :17 tag.

The root cause of the story is not Kyverno 1.19. It is an update put off until it was two releases at once, tested against a version the cluster did not run.

Containment

Roll back through Git, not by hand. The version is a line in Git; reverting the pull request brings the old chart back the same way it left. A helm rollback by hand is undone by Flux at the next reconcile.
Fix forward when the cause is understood. Here, the generated controller rules were turned off - one line, its reason written next to it - instead of staying on the old version.
Make the tests run what runs. The tool-versions check fails the pull request that bumps a chart without its tool, or the other way round.
Give slow steps their time, and only those. The Helm timeout went to 15 minutes - image pulls take as long as they take. The webhook timeout stayed: the fix was to stop verifying Deployments, not to wait longer for it.

Updates in small steps, on a schedule we keep, tested against what runs - a version that is not current is debt that grows every month.

Pause and Predict: A pull request bumped a chart and was merged. flux get helmreleases still shows the old chart version, READY is False, and the message mentions a rollback. Which version runs - and where do you look first?