The Incident: The Load Test That Hit Production

Incident Hook

Thursday, 15:30. Boris wants to know how the backend in develop behaves under load. He starts a load-test pod in develop and points it at the backend - with a URL he copied from a production runbook a minute earlier:

http://backend.production.svc.cluster.local/api/...     # meant: backend.develop

2,000 requests a second, for ten minutes. In this story the cluster has no NetworkPolicies yet, and nothing between develop and production says no.

At 15:34 the production backend scales to its maximum. At 15:38 the latency alert fires: real users wait four seconds per page. Nobody deployed anything. Flux shows no change. The production dashboards show a flood of requests that did not come through the ingress at all.

Result: a test in develop became an outage in production - through a path that nobody needed and nobody had closed.

The story is invented, and it is set on a cluster without the platform’s network baseline. The names and paths are this platform’s: the lab shows what the same pod gets today.

Observed Symptoms

What on-call sees at 15:38:

production backend:  HPA at maximum replicas, p99 latency 4.1 s
ingress (Traefik):   request rate normal
source of the extra requests: 10.244.1.37 - a pod IP, not the ingress

The extra load does not come through the front door. It comes from inside the cluster.

Confusion Phase

Everything the team usually checks is clean:

• no deploy, no merge, no Flux change in production;
• the ingress shows normal traffic;
• production is read-only for every everyday sign-in (Chapter 01) - nobody changed anything there.

So how does a pod that nobody may deploy to production reach production?

What AI Would Propose (Brave Junior)

• “Tell the team to double-check URLs before load tests.”
• “Rate-limit the backend, then it does not matter who calls it.”
• “Put production in its own cluster - until then, nothing can be done.”

Why This Is Dangerous

• Care is not a control. The next copied URL, the next misconfigured client, the next compromised pod is a matter of time. Chapter 01’s rule holds here too: make the wrong path impossible, not merely unlikely.
• A rate limit hides the path, it does not close it. The load test would have been slower - and still in production. And a rate limit does nothing against a pod that reads data instead of flooding.
• RBAC is not network isolation. Chapter 01 stopped people from changing production. It does not stop a pod from connecting to it: permissions decide who may call the Kubernetes API, NetworkPolicies decide which pod may open a connection to which.
• Separate clusters are a valid answer - and not the only one. Inside one cluster, policies already make develop and production separate networks for the paths that matter.

Pause and Predict: Which two policies would have stopped the load test - one on the develop side, one on the production side? And which of the two alone would have been enough?