Investigation & Containment

Investigation

Read the evidence before you change a number. Every command names the cluster with --context (Chapter 01); on your kind cluster the same commands answer for your own develop.

  1. Why did the container stop? Not the events - the container’s status:

    kubectl --context kind-sre-control-plane -n develop get pods -l app=backend \
  -o custom-columns='POD:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason,EXIT:.status.containerStatuses[0].lastState.terminated.exitCode'

    On your cluster: RESTARTS 0, LAST <none> - nothing has died. In the story: OOMKilled and 137. The reason is in lastState because the container was restarted; the kubelet’s events only say Back-off restarting failed container. Exit code 137 alone is not proof: any SIGKILL gives 137. OOMKilled says the kernel’s OOM killer ended it - usually at the container’s own limit, but not always: a node that runs out of memory before the kubelet can evict anything gets the kernel’s node-wide OOM killer, and its victim shows OOMKilled too. Step 4 tells the two apart.

  2. What did it ask for, and what class is it?

    kubectl --context kind-sre-control-plane -n develop get pods -l app=backend \
  -o custom-columns='POD:.metadata.name,QOS:.status.qosClass,MEM_REQ:.spec.containers[0].resources.requests.memory,MEM_LIM:.spec.containers[0].resources.limits.memory,CPU_REQ:.spec.containers[0].resources.requests.cpu,CPU_LIM:.spec.containers[0].resources.limits.cpu'

    Burstable, 64Mi requested, 256Mi limit - develop’s values; staging and production run with 128Mi and the same 256Mi limit. The limit is where the kernel kills it.

  3. What does it really use?

    kubectl --context kind-sre-control-plane -n develop top pod -l app=backend

    One number, now - around 20Mi on an idle kind cluster. kubectl top shows the working set: an estimate of the memory the container cannot easily give back - its total use minus the file cache the kernel can reclaim. The limit is enforced on the container’s whole cgroup (memory.current against memory.max), and the kernel reclaims cache before it kills, so do not expect the kill exactly when top reaches the limit; the kubelet uses the working set to rank pods when the node runs short. Read it as an estimate and a trend. A single reading says nothing about a trend; the question “a size or a leak?” needs the same number over time - the memory graph in Grafana (Chapter 11), or kubectl top repeated under the same load.

  4. Is it the container - or the node? OOMKilled alone does not say which: the kernel’s OOM killer leaves the same reason at the container’s own limit and when the whole node runs out. The node holds the evidence - its conditions, the pods it evicted, and a SystemOOM event:

    kubectl --context kind-sre-control-plane get nodes \
  -o custom-columns='NODE:.metadata.name,MEMORY_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status,DISK_PRESSURE:.status.conditions[?(@.type=="DiskPressure")].status'
kubectl --context kind-sre-control-plane get pods -A --field-selector status.phase=Failed

    False, False and No resources found. An evicted pod would be listed here as Failed, with Evicted in its status - a kubelet decision for the node, not the kernel’s for one container. When the node ran out before the kubelet could evict, the kubelet records a SystemOOM event on the node instead:

    kubectl --context kind-sre-control-plane get events -A --field-selector reason=SystemOOM

    No resources found - no node-wide OOM. An OOMKilled container with no SystemOOM at that time, and a working set near its limit, hit its own limit; with a SystemOOM, the node did.

  5. What does the namespace allow?

    kubectl --context kind-sre-control-plane -n develop describe resourcequota compute-quota
kubectl --context kind-sre-control-plane -n develop describe limitrange default-container-limits

    The quota: Used against Hard for the namespace’s total - requests.memory 288Mi of 1Gi. The LimitRange: per container, Min 32Mi, Max 512Mi, and the defaults a container without numbers would get. In the story, 1Gi per container was over the Max - refused at pod creation, whatever the quota said.

  6. Would the change fit? In your clone of the platform repository - the check the pull request ran:

    scripts/check-app-resources.sh

    One line per namespace - the worst case against the quota - and check-app-resources: OK. With the story’s 1Gi it printed the two problems instead.

The root cause is not the limit and not the quota. It is memory that grows without end: the limit only made it visible - in one container, early, instead of on the whole node, later.

Containment

  1. Roll back the release in Git. Flux brings the previous image back (Chapter 03); the restarts stop. The limit is not touched.
  2. Find the growth. A working set that levels off is a size; one that climbs from the release on is a leak. Fix the code - here, a cache with a maximum size.
  3. Size from evidence, in develop first: the working set under realistic load, plus headroom. If the limit really must change, it changes in staging and production together, and check-app-resources.sh checks the worst case before the merge (Workflow).

A bigger number hides a leak; it does not fix it. Read the container’s status and the trend first - change a limit only for a size you have measured.

Pause and Predict: The quota in production shows requests.cpu Used 260m of Hard 1, and kubectl top shows the pods using far less. A pull request raises the backend’s CPU request from 100m to 250m - two pods would need only 300m more, and the namespace has room. Will check-app-resources.sh let it through? (It counts the backend at its HPA maximum plus the rolling-update surge.)