Investigation & Containment

Investigation

Start from the traffic, not from the code: nothing was deployed, so the question is who connects, and what lets them. Every command names the cluster with --context (Chapter 01) - here the team’s Hetzner cluster; on your kind cluster the same commands run with kind-sre-control-plane.

Who sends the extra requests? The source in the backend’s logs is a pod IP. Find the pod:
```
kubectl --context hetzner-sre-control-plane get pods -A -o wide | grep 10.244.1.37
```
In the story it prints develop loadtest-boris 1/1 Running ... - a pod in another environment. (The IP and the pod are the scenario’s; on your own cluster, use a source IP from your backend’s logs.)
What controls the path into the production backend? The policies that select it:
```
kubectl --context hetzner-sre-control-plane -n production get networkpolicy
```
In the story: No resources found in production namespace. No ingress policy selects the backend, so nothing on the backend’s side refuses a connection - only a policy on the source’s side could still stop one, and develop had none either. On the platform today the list starts with default-deny-all, and allow-backend-ingress names exactly two sources - pods labelled app=frontend (or frontend-primary, Flagger’s copy, Chapter 19) in production, and the traefik namespace - both on TCP 8080.
Is the path really open - and is it the rules, or something else? Probe it from where the load test ran, and read both answers. scripts/np-probe.sh -n <namespace> [-c <context>] [-l key=value] <host> <port> - -c selects the cluster, by default $KUBE_CONTEXT, else kind-sre-control-plane:
```
scripts/np-probe.sh -c hetzner-sre-control-plane -n develop backend.production.svc.cluster.local 80
```
- DNS ok (...), TCP open - the name resolves and nothing stops the connection: the path is open.
- DNS ok (...), TCP blocked - the name resolves, the connection does not: something stops it. With the platform’s policies, that is the answer today.
- DNS FAILED - the lookup failed. Either the name is wrong (a typo, a Service that does not exist), or the probe pod may not reach CoreDNS: with egress isolated and no DNS allow that matches, the lookup itself is blocked. Check both - the name, and the source’s DNS egress rules. On this platform every environment has allow-dns-egress; on a cluster without it, a default deny ends here.
A bare nc ... && echo OPEN || echo BLOCKED cannot tell the last two apart. That is why the probe asks DNS first.

The root cause is not Boris’s typo. It is a flat network: with no policy on either side, a pod in develop could open a connection to the production backend - and, for the same reason, to anything else in the cluster. The only thing between develop and production was attention.

Containment

Stop the traffic now. Boris deletes his own pod: kubectl --context hetzner-sre-control-plane -n develop delete pod loadtest-boris. That ends this incident - not the next one.
Close the path in Git, through a pull request (Chapter 05): a default-deny-all per environment, and one allow policy for each path the platform needs. They go into flux/infrastructure/network-policies/base, which all three environments build on - so read flux diff for each of them before the merge (Chapter 03).

Prove it from both sides after the merge: the probe from step 3 now prints TCP blocked, and the paths that must work still do. In develop - the same base as production, and a namespace the everyday sign-in may create pods in:

scripts/np-probe.sh -c hetzner-sre-control-plane -n develop -l app=frontend backend 80             # TCP open
scripts/np-probe.sh -c hetzner-sre-control-plane -n develop -l app=backend app-postgres-rw 5432    # TCP open

A path nobody needs is a path nobody should have. In each environment the default is nothing; every connection that exists is written down - and a probe proves it.

Pause and Predict: You add the default-deny-all to production and nothing else. What happens to real users within the next minute - and which policy should have gone in first?