The Incident: The Limit That Kept Growing

Incident Hook

Tuesday, 10:05. Ana’s backend release adds a small in-memory cache for the slowest API responses. In develop - one pod, little traffic - it is fast and quiet. At 10:20 the new image reaches staging.

11:10. The QA team’s smoke run gets 502s from staging. The backend pods show RESTARTS 3, then 4. kubectl describe pod says why the last one died:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137

11:20. Boris asks his AI agent to fix it. The agent proposes the obvious: the pods need more memory - raise the limit from 256Mi to 1Gi. Boris changes the base, so staging and production stay equal, and opens a pull request. The hooks check fails:

check-app-resources: 2 problem(s):
  - staging: worst case (backend 3x frontend 3x postgres 1x barman-sidecar 1x) needs limits.memory (Mi)=4608 but the quota allows 3072
  - production: worst case (backend 4x frontend 4x postgres 2x barman-sidecar 2x) needs limits.memory (Mi)=6656 but the quota allows 4096

11:35. The agent has the next fix ready: raise both quotas to 7Gi. Ana stops it and opens the memory graph of the staging backend instead. Since 10:20 its working set has climbed in a straight line - 60Mi, 120Mi, 200Mi, 256Mi, dead, again. It never levels off. The cache has no size limit: every new response stays in memory forever. At 1Gi the pods would have died too - later, after taking four times as much from the node.

And the quota was not the only wall: the namespace’s LimitRange allows at most 512Mi per container. Even with a bigger quota, every new pod would have been refused at creation.

11:50. Ana rolls staging back to the previous backend image in Git. The restarts stop. In the afternoon the cache gets a maximum size; under the same load in develop the working set settles at 140Mi. The limit stays 256Mi.

Result: ninety minutes of errors in staging, and two “fixes” that would have hidden a memory leak behind a bigger number - stopped by a check that knew the budget.

The story is invented; the numbers, the messages and the checks are this platform’s.

Observed Symptoms

• kubectl get pods: the backend’s RESTARTS climbing; in between, CrashLoopBackOff.
• kubectl describe pod: Last State: Terminated, Reason: OOMKilled, Exit Code: 137 - 128 + 9, the signal SIGKILL. The application did not exit; the kernel killed it.
• The events say little: Back-off restarting failed container. The reason is in the container’s status, not in the events.
• The memory graph: a straight line up from the moment of the release.

Confusion Phase

The error looks like a sizing problem:

• the pods die at exactly their limit - so the limit must be too low;
• the release only added a cache - caches use memory, that is their job;
• develop was fine - so staging must be configured too tightly.

Is the limit too low - or is it the only thing that made the leak visible?

What AI Would Propose (Brave Junior)

• “Raise the limit to 1Gi - the app clearly needs more.”
• “Remove the memory limit, so it can use what it needs.”
• “The quota blocks the change - raise it.”

Why This Is Dangerous

• A leak has no right size. A container whose memory grows without end reaches every limit; a bigger one only moves the crash later and makes it bigger.
• Without a limit, the leak becomes the node’s problem. The container takes memory from every pod beside it, until the node runs short and the kubelet starts evicting pods - other teams’ pods among them.
• The quota is a budget, not an obstacle. It is the share of the cluster the namespace may promise. Raising it to fit one change promises memory the nodes may not have.
• The evidence was one graph away. A working set that levels off is a size; a straight line up is a leak.

Pause and Predict: The pods died with OOMKilled and exit code 137. Who killed them - Kubernetes, the kubelet, or the kernel? And why does the reason appear in the container’s status and not in the events?