Investigation & Containment

Investigation

Read the error before you change a setting. Every command names the cluster with --context (Chapter 01); on your kind cluster the same commands answer for your own develop.

  1. Why does the pod stop? Its last log lines:

    kubectl --context kind-sre-control-plane -n develop get pods -l app=frontend
POD=$(kubectl --context kind-sre-control-plane -n develop get pods -l app=frontend \
  -o jsonpath='{range .items[?(@.status.containerStatuses[0].state.waiting.reason=="CrashLoopBackOff")]}{.metadata.name}{"\n"}{end}' | head -1)
[ -n "$POD" ] && kubectl --context kind-sre-control-plane -n develop logs "$POD" --previous --tail=5 \
  || echo "no frontend pod in CrashLoopBackOff"

    The second command picks a pod that is in CrashLoopBackOff; --previous reads the log of its last, failed run. On your cluster no frontend pod crashes, so it prints no frontend pod in CrashLoopBackOff. In the story: mkdir() "/var/lib/nginx/proxy-cache" failed (30: Read-only file system). Error 30 is EROFS: the write went to the root filesystem, which is read-only. It is not a permission problem - a Permission denied (error 13, EACCES) would point at file ownership instead (Workflow).

  2. Which paths may the container write? Exactly its mounted volumes:

    kubectl --context kind-sre-control-plane -n develop get deploy frontend \
  -o jsonpath='{range .spec.template.spec.containers[0].volumeMounts[*]}{.mountPath}{"\n"}{end}'

    /var/cache/nginx, /var/run, /tmp - and not /var/lib/nginx. The new configuration writes to a path nobody declared.

  3. Why does a patched Deployment start no pods? Pod Security checks pods. A Deployment with root is accepted with a warning; the ReplicaSet then tries to create the pods, and that is refused:

    kubectl --context kind-sre-control-plane -n develop get events --field-selector reason=FailedCreate

    Error creating: pods ... is forbidden: violates PodSecurity "restricted:latest" ... - on the ReplicaSet, not on the Deployment. On your cluster the list is empty: nothing has tried.

  4. What does the running container actually have? Not the manifest - the pod:

    kubectl --context kind-sre-control-plane -n develop exec deploy/frontend -- id
kubectl --context kind-sre-control-plane -n develop get pod -l app=frontend \
  -o jsonpath='{.items[0].spec.securityContext}{"\n"}{.items[0].spec.containers[0].securityContext}{"\n"}'

    uid=10001 - not root; the pod’s context with runAsNonRoot and seccompProfile: RuntimeDefault, the container’s with readOnlyRootFilesystem: true, allowPrivilegeEscalation: false and drop: ["ALL"]. A setting is a promise; the running pod is the proof.

The root cause is not nginx and not the hardening. It is a writable path nobody declared: the change needed one directory, and asked for everything instead.

Containment

  1. Undo the hand patch. Flux sets the Deployment back at its next reconcile (Chapter 03); a flux reconcile does it now. The old pods never stopped serving.
  2. Declare the path in Git, through a pull request (Chapter 05): an emptyDir mounted at /var/lib/nginx, with a sizeLimit. hooks checks it - non-root, read-only root, every emptyDir limited.
  3. Prove it after the merge: the pods start, id still prints uid=10001, and a write outside the declared paths still fails with Read-only file system.

Give the app the one path it needs - never the whole container. A writable path is declared, mounted and limited; everything else stays read-only.

Pause and Predict: Ana’s volume has a sizeLimit of 64Mi. What happens on the day nginx writes 65Mi into it - to the pod, and to the other pods on the same node?