Workflow: A Budget, Not a Guess

Two Numbers, Three Readers

Every app container on the platform names a request and a limit for CPU and memory. Three different parts of the system read them - and each reads something else:

	Request	Limit
The scheduler	places the pod on a node where the sum of requests still fits - what pods use does not count	-
The quota	counts `requests.cpu`, `requests.memory` against the namespace’s total	counts `limits.cpu`, `limits.memory` against the namespace’s total
The kernel, CPU	the container’s share when the CPU is contended	a hard ceiling on time: over it, the container is throttled - paused until the next 100ms period
The kernel, memory	the line the kubelet measures against when the node runs short	a hard ceiling: over it, the kernel’s OOM killer kills the process - `OOMKilled`, exit code 137

Why CPU and memory end differently. CPU time can be withheld and given back a moment later: a throttled container only gets slower. Memory cannot be taken back from a process that holds it - so the only way to enforce a memory limit is to kill the process.

Who Stopped the Container

Five different mechanisms stop a container or keep it from starting. Each leaves its trace somewhere else - and the first step of every investigation is to know which one acted:

What happened	Who decides	What you see	Where
OOMKilled	the kernel: at the container’s memory limit - or, when the whole node runs out before the kubelet can evict, the node-wide OOM killer	`Reason: OOMKilled`, `Exit Code: 137`; restarts - for both	the container’s status (`state` / `lastState.terminated`); a node-wide kill also leaves a `SystemOOM` event on the node
Evicted	the kubelet, when the node runs short of memory or disk	the pod `Failed` with `Reason: Evicted`; the node with `MemoryPressure` / `DiskPressure`	the pod’s status, the node’s conditions
Throttled	the kernel, at the container’s CPU limit	nothing - only slower responses	the CPU metrics - `container_cpu_cfs_throttled_periods_total`, which Chapter 11 reads in Prometheus; the lab reads the same counter directly, from `cpu.stat` in the container’s cgroup
Refused by the quota	the API server, when the pod is created	the Deployment accepted, fewer pods ready than asked; `FailedCreate ... exceeded quota`	the ReplicaSet’s events
Refused by the LimitRange	the API server, when the pod is created	`forbidden: maximum memory usage per Container is 512Mi`	the ReplicaSet’s events - or straight back from `kubectl apply` for a bare pod

The last two are the same pattern as Pod Security in Chapter 07: the quota and the LimitRange check pods, not Deployments. A Deployment over the quota is accepted - and its ReplicaSet then fails to create every pod.

QoS Classes: What They Decide - and What They Do Not

Kubernetes gives every pod a Quality of Service class, computed from its numbers:

Guaranteed - every container has CPU and memory limits, and its requests equal them.
Burstable - not Guaranteed, but at least one container has a CPU or memory request or limit.
BestEffort - no container has any request or limit.

When the node runs short of memory, the kubelet does not sort pods by class. It ranks them by three things, in order: whether a pod uses more memory than it requested, then its priority (PriorityClass), then how far over its request it is. The class follows from that: a BestEffort pod requested nothing, so any use is over - it comes first; a Guaranteed pod cannot use more than it requested - it is never in the first group. But a Burstable pod below its requests sits in the same group as Guaranteed pods, and a higher priority can save a pod a lower-priority Guaranteed pod is not saved by. If the kernel runs out of memory before the kubelet acts, its node-wide OOM killer picks by a score in which the class does count: BestEffort first, Guaranteed last. Under disk pressure the same three steps apply to ephemeral-storage use and requests instead - and the QoS class, computed from CPU and memory only, says nothing there.

What the class does not decide:

Scheduling priority. Which pod gets a place, or preempts another, is Pod Priority (PriorityClass) - a separate setting, and one that also outranks the class when the kubelet evicts.
Protection from the container’s own limit. A Guaranteed container that goes over its memory limit is OOMKilled like any other. The lab kills one of each.

The platform’s choice: Burstable. The backend, the frontend and Postgres request what they use most of the time and are allowed more: requests are small, so they fit small nodes; limits leave room for a burst. The price is honest: when a node runs short, a Burstable pod that is over its request is an early candidate for eviction. On a larger cluster, a database is often Guaranteed - it reserves its burst all the time, and is the last pod to be evicted. Here, the nodes are small and the platform pays with that risk. “Guaranteed is safest” is a trade-off, not a rule.

The Platform’s Budget

Per container: the LimitRange - the same in develop, staging and production:

	Min	Max	Default request	Default limit
CPU	10m	500m	10m	250m
memory	32Mi	512Mi	64Mi	256Mi

A container outside min/max is refused. A container without numbers is not refused: it silently gets the defaults - which is why the check in the pull request refuses a missing request or limit instead of relying on them.

develop LimitRange

Show the develop LimitRange

apiVersion: v1
kind: LimitRange
metadata:
  name: default-container-limits
  namespace: develop
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 10m
        memory: 64Mi
      default:
        cpu: 250m
        memory: 256Mi
      min:
        cpu: 10m
        memory: 32Mi
      max:
        cpu: 500m
        memory: 512Mi

Per namespace: the ResourceQuota.

Namespace	Requests (CPU / memory)	Limits (CPU / memory)	Pods
`develop`	1 / 1Gi	2 / 2Gi	15
`staging`	1 / 1Gi	3 / 3Gi	15
`production`	1 / 1.5Gi	3 / 4Gi	15

The limits add up to more than the requests: the platform deliberately promises bursts that not every pod will take at once. Staging equals production for the applications because staging is production’s rehearsal: with more room, a release could fit staging and still be refused in production. Production’s quota is larger only by its database: production runs it as two instances, a primary and a replica, and the second one with its backup sidecar needs its own requests and limits.

develop ResourceQuota

Show the develop ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: develop
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
    pods: "15"

The apps’ numbers. The backend’s base Deployment holds the production values; staging uses them as they are, develop patches in smaller ones. Every number is in Git and changes through a pull request.

Backend Deployment (resources)

Show the backend Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
    app.kubernetes.io/name: backend
    app.kubernetes.io/component: api
spec:
  # No spec.replicas: the HorizontalPodAutoscaler owns the replica count (hpa.yaml per environment,
  # minReplicas is the floor). A replicas value in Git makes Flux reset what the HPA scaled on every
  # reconcile. Existing clusters: run scripts/hpa-replicas-handover.sh once before this change.
  revisionHistoryLimit: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
        app.kubernetes.io/name: backend
        app.kubernetes.io/component: api
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      # No <SERVICE>_PORT=tcp://... variables for the namespace's Services; KUBERNETES_SERVICE_* stay
      # (scripts/check-app-security.sh says why).
      enableServiceLinks: false
      # The backend never calls the Kubernetes API: no ServiceAccount token in the pod.
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      # Schema migrations before the app (Chapter 18): the same image reference as the app (the
      # overlay's images: transformer sets both tags; Kyverno verifies the digest each tag resolves to
      # at admission - a tag is not pinned, see Chapter 17) runs `backend migrate`, applies the migrations
      # embedded in it that the database lacks (golang-migrate, under a Postgres advisory lock) and exits.
      # It runs whenever a backend pod is created - rollout, reschedule, scale-out; with nothing pending
      # it only checks. The recovery point is the continuous WAL archive (Chapter 12), not a backup taken
      # here; a migration's pull request is merged only while that archive is healthy. The app then only
      # checks the schema version and refuses to start on one it cannot use. Migrations only expand:
      # the previous release keeps working on the new schema, so rolling back the image never needs
      # rolling back the data.
      initContainers:
      - name: migrate
        image: backend:latest  # name/tag set by the environment overlay (images: transformer)
        imagePullPolicy: IfNotPresent
        args: ["migrate"]
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          runAsGroup: 10001
          capabilities:
            drop:
              - ALL
        env:
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: app-postgres-app
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-postgres-app
              key: password
        - name: POSTGRES_HOST
          value: app-postgres-rw
        - name: POSTGRES_DB
          value: app
        resources:
          requests:
            cpu: 10m
            memory: 32Mi
          limits:
            cpu: 100m
            memory: 64Mi
      containers:
      - name: backend
        image: backend:latest  # name/tag set by the environment overlay (images: transformer)
        imagePullPolicy: IfNotPresent
        # Stopping, the pod is removed from the Service's endpoints and from Traefik - but not at once,
        # while the container already gets SIGTERM. The sleep keeps it serving until they have caught
        # up; then the app shuts down. Measured on kind (a rollout, ~115 requests/s): without it 14-25
        # failed requests (refused, or 502 from Traefik), with it 0-3 (Chapter 20). The kubelet's own
        # sleep - no shell in the image needed. 5 s + the app's shutdown fit in the 30 s grace period.
        lifecycle:
          preStop:
            sleep:
              seconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 10001
          runAsGroup: 10001
          capabilities:
            drop:
              - ALL
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        env:
        - name: PORT
          value: "8080"
        - name: NAMESPACE
          value: "${NAMESPACE}"
        - name: ENVIRONMENT
          value: "${ENVIRONMENT}"
        - name: LOG_LEVEL
          value: "${LOG_LEVEL}"
        - name: SERVICE_NAME
          value: "backend"
        # No SERVICE_VERSION: the backend reports the version baked into the image (the same as
        # /version), so every span says which release produced it.
        - name: DEPLOYMENT_ENVIRONMENT
          value: "${ENVIRONMENT}"
        # Where a span came from: the cluster, the namespace (= the environment) and the pod - set from
        # the downward API, so traces can be filtered per environment and per pod.
        - name: K8S_NAMESPACE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: K8S_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "k8s.cluster.name=${cluster_name},k8s.namespace.name=$(K8S_NAMESPACE_NAME),k8s.pod.name=$(K8S_POD_NAME)"
        - name: UPTRACE_DSN
          valueFrom:
            secretKeyRef:
              name: backend-secrets
              key: uptrace-dsn
        - name: OTEL_EXPORTER_OTLP_HEADERS
          valueFrom:
            secretKeyRef:
              name: backend-secrets
              key: uptrace-headers
        - name: JWT_SECRET
          valueFrom:
            secretKeyRef:
              name: backend-secrets
              key: jwt-secret
        - name: POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: app-postgres-app
              key: username
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: app-postgres-app
              key: password
        - name: POSTGRES_HOST
          value: app-postgres-rw
        - name: POSTGRES_DB
          value: app
        # Security switches (backend >= the 2026-09-27 hardening). Safe values here; an environment
        # overlay may only turn chaos on (develop, staging). scripts/check-app-security.sh enforces it.
        - name: CHAOS_ENABLED       # /panic + readiness/liveness toggles (still need a token)
          value: "false"
        - name: PPROF_ENABLED       # profiling listener on loopback; never on in Git
          value: "false"
        - name: AUTH_REGISTRATION_ENABLED
          value: "true"
        livenessProbe:
          httpGet:
            path: /livez
            port: http
          initialDelaySeconds: 15
          periodSeconds: 20
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /readyz
            port: http
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 3
          failureThreshold: 3
        startupProbe:
          httpGet:
            path: /healthz
            port: http
          initialDelaySeconds: 0
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 30
        # The production values (staging gets the same; develop patches its own). Requests are what
        # the scheduler reserves and the quota counts; limits are the ceiling: memory over the limit
        # is OOMKilled, CPU over it is throttled. requests < limits makes the pod Burstable.
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
            ephemeral-storage: 64Mi
          limits:
            cpu: 250m
            memory: 256Mi
            ephemeral-storage: 128Mi
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /home/app/.cache
      volumes:
      # Every writable path is an emptyDir with a size limit. It is not a quota that stops the write:
      # the kubelet measures the volume on a schedule and evicts the pod once it is over the limit -
      # a very fast writer can still fill the node's disk before that. It bounds a slow leak and makes
      # the pod's budget explicit. /tmp holds request bodies Go buffers to disk - measured at 0 KB at
      # rest, so 64Mi leaves room for uploads.
      - name: tmp
        emptyDir:
          sizeLimit: 64Mi
      - name: cache
        emptyDir:
          sizeLimit: 10Mi

Before the merge: check-app-resources.sh. It renders every app overlay and fails when:

staging and production differ for an app;
production gets less than develop for any request or limit;
a namespace’s quota cannot hold the worst case: every app at its HPA maximum plus one more pod during a rolling update (maxSurge: 1), plus Postgres - for requests, limits and the pod count.

It runs in pre-commit and again in CI (Chapter 05), and prints the worst case for each namespace even when everything fits. In the incident, it was the first thing that said no.

check-app-resources.sh

Show the resource check

#!/usr/bin/env bash
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${repo_root}"

for tool in kubectl yq; do
  command -v "${tool}" >/dev/null || { echo "check-app-resources: ${tool} is required" >&2; exit 1; }
done

envs=(develop staging production)
apps=(backend frontend)
failures=()

  for app in "${apps[@]}"; do
    rendered="$(get_render "${app}" "${env}")"
    pods="$(worst_case_pods "${rendered}" "${app}/${env}")"
    s="$(pod_resources "${rendered}" "${app}/${env}")"; read -r -a per_pod <<<"${s}"
    for i in 0 1 2 3; do need[i]=$((need[i] + pods * per_pod[i])); done
    need_pods=$((need_pods + pods))
    breakdown+=("${app} ${pods}x")
  done

  # Postgres: every CNPG instance is one more pod with the Cluster's resources.
  cnpg="$(render "flux/infrastructure/data/cnpg-clusters/${env}")"
  instances="$(yqs 'select(.kind == "Cluster") | .spec.instances' <<<"${cnpg}")"
  [[ "${instances}" =~ ^[1-9][0-9]*$ ]] || die "cnpg ${env}: expected one Cluster with a positive spec.instances, got '${instances}'"
  IFS=$'\t' read -r prc prm plc plm <<<"$(yqs 'select(.kind == "Cluster") | .spec.resources | [.requests.cpu, .requests.memory, .limits.cpu, .limits.memory] | @tsv' <<<"${cnpg}")"
  v="$(to_milli "${prc}" "cnpg ${env} requests.cpu")"; need[0]=$((need[0] + instances * v))
  v="$(to_mib "${prm}" "cnpg ${env} requests.memory")"; need[1]=$((need[1] + instances * v))
  v="$(to_milli "${plc}" "cnpg ${env} limits.cpu")"; need[2]=$((need[2] + instances * v))
  v="$(to_mib "${plm}" "cnpg ${env} limits.memory")"; need[3]=$((need[3] + instances * v))
  need_pods=$((need_pods + instances))
  breakdown+=("postgres ${instances}x")
  # The Barman Cloud plugin adds a sidecar to every instance pod, with the resources its ObjectStore
  # names (instanceSidecarConfiguration) - same pod, so the same instance count.
  sidecar="$(yqs 'select(.kind == "ObjectStore") | .spec.instanceSidecarConfiguration.resources | [.requests.cpu, .requests.memory, .limits.cpu, .limits.memory] | @tsv' <<<"${cnpg}")"
  if [[ -n "${sidecar}" ]]; then
    IFS=$'\t' read -r src srm slc slm <<<"${sidecar}"
    v="$(to_milli "${src}" "barman sidecar ${env} requests.cpu")"; need[0]=$((need[0] + instances * v))
    v="$(to_mib "${srm}" "barman sidecar ${env} requests.memory")"; need[1]=$((need[1] + instances * v))
    v="$(to_milli "${slc}" "barman sidecar ${env} limits.cpu")"; need[2]=$((need[2] + instances * v))
    v="$(to_mib "${slm}" "barman sidecar ${env} limits.memory")"; need[3]=$((need[3] + instances * v))
    breakdown+=("barman-sidecar ${instances}x")
  fi

  # What the namespace allows: the ResourceQuota's hard limits.
  quota="$(render "flux/infrastructure/resource-management/${env}")"
  IFS=$'\t' read -r qrc qrm qlc qlm <<<"$(yqs 'select(.kind == "ResourceQuota") | .spec.hard | [."requests.cpu", ."requests.memory", ."limits.cpu", ."limits.memory"] | @tsv' <<<"${quota}")"
  have=()
  v="$(to_milli "${qrc}" "quota ${env} requests.cpu")"; have+=("${v}")
  v="$(to_mib "${qrm}" "quota ${env} requests.memory")"; have+=("${v}")
  v="$(to_milli "${qlc}" "quota ${env} limits.cpu")"; have+=("${v}")
  v="$(to_mib "${qlm}" "quota ${env} limits.memory")"; have+=("${v}")
  for i in 0 1 2 3; do
    if ((need[i] > have[i])); then
      failures+=("${env}: worst case (${breakdown[*]}) needs ${labels[$i]}=${need[$i]} but the quota allows ${have[$i]}")
    fi
  done
  # Pod count: only the apps and Postgres are counted here; other pods in the namespace (CronJobs,
  # lab pods) need the rest of the quota, so this is a floor, not the whole budget.
  quota_pods="$(yqs 'select(.kind == "ResourceQuota") | .spec.hard.pods // ""' <<<"${quota}")"
  if [[ -n "${quota_pods}" ]]; then
    [[ "${quota_pods}" =~ ^[0-9]+$ ]] || die "quota ${env}: pods '${quota_pods}' is not an integer"
    if ((need_pods > quota_pods)); then
      failures+=("${env}: worst case (${breakdown[*]}) needs ${need_pods} pods but the quota allows ${quota_pods}")
    fi
  fi
  echo "  ${env}: worst case (${breakdown[*]}) requests ${need[0]}m/${need[1]}Mi of ${have[0]}m/${have[1]}Mi, limits ${need[2]}m/${need[3]}Mi of ${have[2]}m/${have[3]}Mi, pods ${need_pods} of ${quota_pods:-unlimited}"
done

if ((${#failures[@]} > 0)); then
  echo "check-app-resources: ${#failures[@]} problem(s):" >&2
  printf '  - %s\n' "${failures[@]}" >&2
  exit 1
fi
echo "check-app-resources: OK"

What the Numbers Do Not Do

The quota counts promises, not use. A namespace whose pods use little can still be full - of requests.
kubectl top is one reading. The working set now says nothing about a trend; the question “a size or a leak?” needs it over time.
Throttling leaves no event. A CPU limit set too low shows up only as latency - and in the throttling metrics.
Disk is a budget too. The containers also request and limit ephemeral-storage; the kubelet evicts a pod over that limit, the same way it enforces an emptyDir’s sizeLimit (Chapter 07).
The lab does not push a node. On kind, the node is a container on your laptop that shares its memory: node pressure there would slow down everything you run. The lab kills containers at their own limits, which is safe anywhere.

Guardrails That Stop It

check-app-resources.sh (workstation, CI) - a missing request or limit, staging unequal to production, a worst case over the quota.
LimitRange (cluster) - a container over the max is refused when the pod is created.
ResourceQuota (cluster) - a pod that would push the namespace over its total is refused when it is created.
Flux (cluster) - a number changed by hand is set back at the next reconcile (Chapter 03).
Kyverno require-requests-limits (cluster) - refuses a container without requests and limits in develop, staging and production; elsewhere (the lab namespace) it only reports.

Safe Workflow (Step-by-Step)

Name the mechanism: the container’s status (OOMKilled), the pod’s (Evicted), the ReplicaSet’s events (FailedCreate), or the CPU metrics (throttling).
Read the trend: the working set over time - does it level off, or climb?
Fix the cause if it climbs: a leak is fixed in the code, not with a limit.
Size from evidence if it levels off: measured usage under load plus headroom - in Git, staging and production together.
Pull request: hooks runs check-app-resources.sh; read its worst case.
Prove it after the merge: no restarts, lastState empty, the working set under the limit.

Instructions for Your AI Agent

The rules this chapter adds:

Never remove a request or limit, and never raise a limit to make an OOMKilled pod start. First read the container’s status (reason: OOMKilled) and the memory it really uses. Enforced by: scripts/check-app-resources.sh (a missing request or limit stops it) in pre-commit and CI; Kyverno require-requests-limits refuses it in the application namespaces.
Tell the mechanisms apart before acting: OOMKilled (container status, exit 137), eviction (pod status Evicted, node pressure), throttling (no event - only the CPU metrics). Enforced by: instruction only.
Change staging and production resources together, and never give production less than develop. Enforced by: scripts/check-app-resources.sh.
Never raise a namespace quota to fit a change - show the worst case it has to hold, and let the owner decide. Enforced by: the quotas are in Git (review); scripts/check-app-resources.sh prints the worst case per namespace.

The full list, chapter by chapter, is docs/agent-rules.md in your fork of the platform repository.

This builds on: Security Context (Chapter 07) - after what a container may do comes how much it may use. This enables: Availability Engineering (Chapter 09) - an autoscaler can only add pods the quota can hold.