The Incident: The Hotfix That Wanted Root

Incident Hook

Friday, 17:20. Ana merges a change to the frontend’s nginx configuration: a small cache for API responses, under /var/lib/nginx/proxy-cache. Flux rolls it out to develop. The new pods start - and stop:

nginx: [emerg] mkdir() "/var/lib/nginx/proxy-cache" failed (30: Read-only file system)

CrashLoopBackOff. The old pods still serve, the rollout is stuck, and the demo is at 18:00.

17:35. Boris has the fix in two lines: run the frontend as root and make its filesystem writable. “We’ll do it properly on Monday.” He patches the Deployment in develop by hand:

Warning: would violate PodSecurity "restricted:latest": runAsNonRoot != true ..., runAsUser=0 ...
deployment.apps/frontend patched

Patched. He waits for the new pods. None come. The ReplicaSet’s events say why:

Error creating: pods "frontend-7c9..." is forbidden: violates PodSecurity "restricted:latest": runAsNonRoot != true ...

17:50. He opens a pull request with the same two lines instead. Its hooks check fails:

check-app-security: 2 problem(s):
  - flux/apps/frontend/overlays/develop: Deployment frontend container frontend: runAsNonRoot must be true (got: false)
  - flux/apps/frontend/overlays/develop: Deployment frontend container frontend: readOnlyRootFilesystem must be true (got: false)

18:05. Ana reads the first error again: nginx wants to write one directory. She adds a 64Mi emptyDir at /var/lib/nginx - one volume, no root, nothing else weaker - and the pods start. The demo is late. Nothing else is.

Result: forty-five minutes lost to a fix that would have handed the container to anyone who ever got code running in it - and three layers that refused it.

The story is invented; the settings, the messages and the checks are this platform’s - with one difference: it shows Pod Security alone. On today’s platform a second guardrail stops Boris earlier: Kyverno’s require-security-context (Chapter 16) refuses the Deployment itself in develop, staging and production (admission webhook ... denied the request: Policy require-security-context-enforce failed: Containers must run as non-root ...), so his hand patch would not even have been saved.

Observed Symptoms

• kubectl get pods: the new frontend pods in CrashLoopBackOff, their log ending with (30: Read-only file system) - error number 30 is EROFS.
• After the hand patch: the Deployment says configured, but no new pod appears - the refusal is in the ReplicaSet’s events, not on the Deployment.
• The pull request: the hooks check red, with the exact rule that failed.

Confusion Phase

Under time pressure, the error looks like the platform’s fault:

• the change was a few lines of nginx config;
• “it works on my laptop” - where the container runs as root with a writable filesystem;
• the patch was accepted - so why do no pods start?

Is the hardening in the way - or is it doing its job?

What AI Would Propose (Brave Junior)

• “Set runAsUser: 0 and readOnlyRootFilesystem: false, we’ll harden it later.”
• “Add privileged: true so it can do whatever it needs.”
• “Change the namespace’s Pod Security label to baseline for the weekend.”

Why This Is Dangerous

• “Later” is never measured. The weaker setting stays until someone remembers it - and every vulnerability found in the meantime meets a process that is root, can write to its own binaries, and can install what it lacks.
• privileged: true is not a bigger security context - it is almost none. A privileged container sees the node’s devices and gets every capability; it is the setting closest to the node itself.
• Lowering the namespace’s Pod Security level lowers it for every pod in that namespace, not just for the one that failed.
• The error already said what was missing: one writable directory. The fix is to give the app exactly that.

Pause and Predict: The patched Deployment was accepted, yet no pod started. Where did Pod Security check the root setting - and why there and not on the Deployment?