Workflow: Written-Down Paths

What NetworkPolicies Give You

• A small blast radius. When one pod goes wrong - a mistake, a bug, a compromised dependency - it can reach only the paths written down for it. The frontend’s pod can call the backend, and nothing else; if it is taken over, the databases are still out of reach.
• Environments that are separate networks. develop, staging and production share one cluster but not their direct paths: no pod in develop opens a connection to a pod in production - the incident shows the hard way what happens without that. They are not sealed off from each other: the backend in develop may still reach the public production site over HTTPS, through Cloudflare, like anyone on the internet.
• Less to leak. A pod that may open connections only where it needs to cannot send data anywhere else either.
• A map that is true. The policies in Git are the list of who talks to whom, on which port, reviewed like every other change - and the cluster enforces them. Unlike a diagram, they cannot go out of date - as long as nobody adds a policy by hand: one that is not in Git adds allowances the map does not show (Guardrails That Stop It).
• A second layer after RBAC. Chapter 01 decided who may change what. NetworkPolicies decide which pod may connect to which - a different question, asked on every connection.

The price: a new path needs a policy before it works, and a missing one breaks a working path. That failure is loud and local - a probe finds it in minutes. The failure the policies prevent is silent and cluster-wide.

How a NetworkPolicy Works

A NetworkPolicy lives in one namespace and selects pods there by their labels (podSelector). For those pods it lists what is allowed, per direction:

• Ingress - who may connect to them, and on which ports.
• Egress - where they may connect to.

Five rules explain everything the lab shows:

1. No policy, no isolation. A pod that no policy selects accepts and opens every connection. That is the flat network.
2. Selected means isolated - in that direction. As soon as any policy selects a pod for ingress, the pod accepts only what some policy allows. The same for egress. default-deny-all simply selects every pod (podSelector: {}) for both directions and allows nothing.
3. Policies only add. Every policy adds allowances; none can take one away. Two policies are the union of both. So a deny can never “win” over an allow - and one policy that allows everything undoes all the others.
4. Both ends must agree. A connection from an isolated pod to an isolated pod needs the egress allowed on the client and the ingress allowed on the server. The platform writes both: allow-frontend-egress-backend and allow-backend-ingress.
5. The port is the container’s. Policies see the traffic after the Service has forwarded it: the backend’s Service listens on 80, the policy allows 8080.

And three details that cause most surprises:

• podSelector alone means “in this namespace”. allow-backend-ingress allows pods labelled app=frontend - in production only. A pod with the same label in develop is not let in; other namespaces need a namespaceSelector.
• The identity is a label. Any pod labelled app=frontend gets the frontend’s paths. Who may create pods - and so set labels - in a namespace is RBAC’s job (Chapter 01): on Hetzner the everyday OIDC sign-in (members and admins) has the built-in admin role in develop, and nowhere else may create pods. No admission policy restricts the app label, so in develop anyone signed in can start a pod labelled app=frontend - and it gets the frontend’s path to the backend, as A4 of the lab shows. Flux deploys the workloads from Git into all three environments; the break-glass kubeconfig can do anything.
• DNS is a path too. Under a default deny, a pod usually cannot even resolve a name - every connection then fails before it starts. That is why each environment has allow-dns-egress, its own policy, separate from every application rule.

What NetworkPolicies are not. They decide which connection may be opened - nothing more. They do not authenticate the caller, encrypt the traffic or look at a request. They are one layer of a defence, not “zero trust” on their own. The Kubernetes API also records no blocked connection: a blocked packet is simply dropped, so you find it with a probe, not in events or logs.

The network plugin enforces them. A policy the plugin ignores is a comment - kind’s default plugin, kindnet, enforced them only in part. So the platform runs Cilium on every cluster: on kind (kindnet switched off), on Hetzner, and on GKE, whose Dataplane V2 is Cilium. The same policy then behaves the same everywhere - a default deny blocks DNS too, on your laptop as in production (you will see it in the lab). make cluster-conformance checks on each cluster that policies are enforced at all, egress included (its check C2).

Two address families. Every cluster is dual-stack: each pod has an IPv4 and an IPv6 address, and so does each Service that asks for both - the frontend, the backend and Traefik do; the others, the database’s among them, have IPv4 only. A policy that selects pods by label covers both families. An ipBlock covers one family only - 0.0.0.0/0 says nothing about IPv6 - so every IP range comes as a pair: the IPv4 rule and its IPv6 twin. Leave the twin out and the IPv6 path stays closed under the default deny: a host with an IPv6 address times out, now and then. Write ::/0 without cutting out the private IPv6 ranges and you open what the IPv4 rule closed.

The Platform’s Baseline

Every environment namespace builds the same set from flux/infrastructure/network-policies/base:

allow-backend-ingress is the contract for who may call the backend:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-ingress
spec:
  podSelector:
    matchExpressions:
      - key: app
        operator: In
        values: [backend]
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app
                operator: In
                values: [frontend, frontend-primary]
      ports:
        - protocol: TCP
          port: 8080
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: traefik
      ports:
        - protocol: TCP
          port: 8080

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              app.kubernetes.io/name: coredns
        # NodeLocal DNSCache (GKE turns it on): a DNS cache pod on every node answers the kube-dns
        # Service address, so the query never reaches a kube-dns pod. Without this peer nothing resolves
        # under default-deny - nginx in frontend stopped at "host not found in upstream backend" (GKE,
        # 2026-10-15). On clusters without the cache no pod has this label.
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: node-local-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Three rules open every address - deliberately. A NetworkPolicy cannot select a host name, only IP ranges, so a path to something outside the cluster is an ipBlock - one for IPv4 and its twin for IPv6:

• the backend → HTTPS on the internet, for Uptrace - 0.0.0.0/0 with the private and link-local ranges cut out, and ::/0 with their IPv6 counterparts cut out (unique local fc00::/7, link-local fe80::/10, the NAT64 prefixes, and the cluster’s own pod range), so it can never reach anything inside the cluster or the provider’s network;
• the database → HTTPS, for its backups in S3-compatible storage (0.0.0.0/0 and ::/0);
• the database → the API server, which is not a pod and has no fixed address across clusters - on 6443 (kind, k3s) or 443 (GKE, EKS, AKS). This one is written as two halves per family (0.0.0.0/1 + 128.0.0.0/1, ::/1 + 8000::/1): Cilium reads exactly 0.0.0.0/0 as “outside the cluster”, which leaves out the nodes - and on kind and k3s the API server runs on a node. The halves reach a node only because the platform sets Cilium’s policyCIDRMatchMode: nodes (by default an IP rule never matches a node); conformance check C3 proves the path on each cluster.

Each is a trade-off with a remaining risk, written next to the rule in the repository: the backend and the database could send data to any public HTTPS host. Narrowing them needs a stable address range - or an egress proxy, which this platform does not run. “Never 0.0.0.0/0” is not the rule; “never without a port, a reason and the remaining risk written down” is.

One Base, Three Environments

The three environments build on the same base (Chapter 03): a policy changed there changes develop, staging and production at the next reconcile. There is no staged rollout of a base change. So:

• read flux diff for each environment before the merge, and for every deleted policy ask what it did: an allow policy that goes closes a path; a policy whose selector isolated pods - a default deny, or the only policy selecting some pods - opens them up instead. Read its podSelector and policyTypes, and what remains;
• for a change that should start in develop alone, put it into the develop overlay first, and move it into base once it is proven.

Guardrails That Stop It

• The default deny (cluster). A path that nobody wrote down does not exist - including the one from the incident.
• Flux (cluster). A policy it manages is set back at the next reconcile when someone changes or deletes it by hand (Chapter 03). It does not remove a policy someone added by hand: Flux deletes only what it created. An “allow everything” policy applied with kubectl stays until someone deletes it - verified on the kind cluster.
• The checks on every pull request (CI). Flux Diff builds the policies and validates them against the schema (Chapter 05). scripts/check-app-security.sh refuses a rule that opens every address (/0, or the /1 halves) in one family without the other, and an IPv6 twin that leaves open the private ranges its IPv4 rule cuts out. A narrower range without its twin passes - that one is the reviewer’s. Whether a rule is too wide otherwise is a reviewer’s question - no check answers it.
• RBAC (cluster). On Hetzner the everyday OIDC sign-in may write only in develop (Chapter 01): with it, nobody adds an “allow everything” to production by hand. The break-glass kubeconfig could - which is why it is kept for emergencies.

Triage: a Connection Is Blocked

1. DNS first. Probe from the source’s namespace, with the source’s labels, to the target’s name and port - for example the frontend to the backend in develop: scripts/np-probe.sh -n develop -l app=frontend backend 80. Replace the namespace, the labels (one -l key=value per label), the name and the port with your own. On DNS FAILED, check two things before anything else: is the name right (does the Service exist), and may the source reach CoreDNS - is it egress-isolated without a DNS allow that matches its labels? Only with DNS ok go on to the TCP path. A connection to a dual-stack Service that fails only now and then is often one family: probe each with -4 and -6. Only there - for a Service with IPv4 only, -6 finds no IPv6 address and prints DNS FAILED, which then says nothing about policies (kubectl get service <name> -o jsonpath='{.spec.ipFamilies}' shows the families).

2. Which policies select the target? Name the two namespaces once - for the incident’s path, the source in develop and the target in production:

   SOURCE_NS=develop TARGET_NS=production
   kubectl --context kind-sre-control-plane -n "$TARGET_NS" get networkpolicy -o yaml
   

   Then read each one whose podSelector matches the target’s labels: is there an ingress rule for the source - its labels, its namespace, the container port?

3. Which policies select the source? The same for egress on the source’s side: kubectl --context kind-sre-control-plane -n "$SOURCE_NS" get networkpolicy -o yaml.

4. Labels and namespaces. kubectl --context kind-sre-control-plane -n "$SOURCE_NS" get pods --show-labels and the same with "$TARGET_NS"; for a namespaceSelector, the namespace’s labels.

5. Fix the exact pair in Git - and prove it with the probe after the merge.

Safe Workflow (Step-by-Step)

1. Write down the path: source labels and namespace, target labels and namespace, container port.
2. Write both ends: egress for the source, ingress for the target - in the overlay for one environment, or in base for all three.
3. Preview: flux --context kind-sre-control-plane diff kustomization network-policies-develop --path ./flux/infrastructure/network-policies/develop, and the same with staging and production for every environment the change reaches.
4. Merge through a pull request (Chapter 05).
5. Prove it: the new path open with the right labels, still blocked without them - with scripts/np-probe.sh.

Instructions for Your AI Agent

The rules this chapter adds:

1. Change NetworkPolicies only in Git, through a pull request; read flux diff for every environment the base feeds. Never apply, edit or delete one in an environment namespace by hand. Enforced by: on Hetzner the OIDC roles may write only in develop. Flux sets back a changed or deleted policy it manages - not one created by hand, which stays: instruction only.
2. To fix a blocked connection, propose the exact pair - source labels, target labels, port, both directions - never an empty from/to, a podSelector: {} without ports, or a new 0.0.0.0/0 rule. Enforced by: the review of the pull request; no check refuses them - instruction only.
3. Diagnose before proposing: run scripts/np-probe.sh (DNS and TCP separately) and report which policies select the source and the target. Enforced by: instruction only.
4. Never give a pod the labels of another application to get through a policy. Enforced by: instruction only; on Hetzner the everyday OIDC sign-in can create pods only in develop.
5. Experiments go into the namespace lab, and what you created there is deleted afterwards. Enforced by: instruction only.
6. Every IP range in a policy comes with its twin of the other family - 0.0.0.0/0 with ::/0, and the private IPv4 ranges cut out mean fc00::/7, fe80::/10, ::1/128, the NAT64 prefixes and the cluster’s own pod range (${cluster_ipv6_range} - global on GKE) cut out of ::/0. Only a rule that must reach a node uses the /1 halves. Never switch IPv6 off or leave a twin out to make a policy look safe. Enforced by: scripts/check-app-security.sh in pre-commit and CI, only for the rules that open every address (/0 or a /1 half): one in IPv4 needs one in IPv6 and the reverse - their forms are not compared - and an IPv4 rule that cuts out the private ranges needs ::/0 with fc00::/7, fe80::/10 and the NAT64 prefixes cut out. Matching forms, narrower twins, ::1, the pod range and where the /1 halves are used - instruction only.

The full list, chapter by chapter, is docs/agent-rules.md in your fork of the platform repository.

This builds on: CI/CD & Developer Guardrails (Chapter 05) - every policy change reaches the cluster through a pull request whose checks you can read. This enables: Security Context (Chapter 07) - the network limits where a pod may connect; the security context limits what it may do inside.