Workflow: Count, Don't Hope

Who Owns the Replica Count

The HPA changes a Deployment’s spec.replicas between its minReplicas and maxReplicas. Flux applies the same Deployment from Git every few minutes and sets back every field it manages (Chapter 03). If both wrote spec.replicas, they would fight: the HPA scales to 3 in a peak, Flux sets it back to the 2 in Git, the HPA scales up again - over and over, during exactly the minutes that matter.

So on this platform the HPA alone owns the count:

• No spec.replicas in Git for any Deployment an HPA scales. Flux applies with server-side apply: a field it has never set, it leaves alone; in the cluster, the field’s owner is the HPA (kube-controller-manager, through the scale subresource) - the lab shows it. A field Flux did set is different - see the handover below.
• check-app-security.sh refuses it in the pull request: Deployment backend has an HPA and must not set spec.replicas.
• An existing cluster needs one handover when the field leaves Git: if Flux was its only owner, dropping the field from Git makes the API server remove it - and the count falls back to the default of 1. hpa-replicas-handover.sh first makes another field manager an owner; after that, Flux gives up the field without resetting it and leaves it to the HPA. A new cluster does not need it: its Deployments never had the field in Git.

#!/usr/bin/env bash
context="${1:?usage: $0 KUBE_CONTEXT [--apply]}"
mode="${2:-dry-run}"
if [[ "${mode}" != "dry-run" && "${mode}" != "--apply" ]]; then
  echo "error: unknown argument '${mode}' - use nothing (dry run) or --apply" >&2
  exit 2
fi
namespaces=(develop staging production)
  # Captured, not a process substitution: a failed lookup must stop the run, not look like "no HPAs".
  if ! hpas="$(kube -n "${ns}" get hpa -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.scaleTargetRef.kind}{"\t"}{.spec.scaleTargetRef.name}{"\n"}{end}')"; then
    echo "error: cannot list HPAs in ${ns}" >&2
    exit 1
  fi

  for app in "${apps[@]}"; do
    if ! app_deploy="$(kube -n "${ns}" get deployment "${app}" --ignore-not-found -o name)"; then
      echo "error: cannot read deployment ${ns}/${app}" >&2
      exit 1
    fi
    if [[ -n "${app_deploy}" ]] && ! grep -q "Deployment	${app}\$" <<<"${hpas}"; then
      echo "error: ${ns}/${app} exists but no HPA targets it - removing replicas from Git would leave it at 1" >&2
      exit 1
    fi
  done

  while IFS=$'\t' read -r hpa kind target; do
    [[ -z "${hpa}" ]] && continue
    if [[ "${kind}" != "Deployment" ]]; then
      echo "skip ${ns}/${hpa}: targets ${kind}, not a Deployment"
      continue
    fi
    if ! replicas="$(kube -n "${ns}" get deployment "${target}" --ignore-not-found -o jsonpath='{.spec.replicas}')"; then
      echo "error: cannot read deployment ${ns}/${target}" >&2
      exit 1
    fi
    if [[ -z "${replicas}" ]]; then
      echo "skip ${ns}/${target}: Deployment not found"
      continue
    fi

    if [[ "${mode}" != "--apply" ]]; then
      echo "would hand over ${ns}/${target} spec.replicas=${replicas} to ${field_manager}"
      continue
    fi
    # A server-side apply of just spec.replicas, with the same value, under our own field manager:
    # the field gets a second owner, so it survives when Flux stops declaring it.
    kube apply --server-side --field-manager="${field_manager}" -f - >/dev/null <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${target}
  namespace: ${ns}
spec:
  replicas: ${replicas}
EOF
    echo "handed over ${ns}/${target} spec.replicas=${replicas} to ${field_manager}"
  done <<<"${hpas}"
done

if [[ "${mode}" != "--apply" ]]; then
  echo "dry run - nothing changed. Re-run with --apply."
fi

What the HPA does - and what it does not. Every 15 seconds it compares the pods’ average CPU and memory with its targets - as a percentage of their requests (Chapter 08) - and sets the count that would bring them to target, within min and max. With two metrics it computes a count for each and takes the larger: if CPU needs two replicas and memory three, it runs three. And because the percentage is of the request, a larger request makes the same usage look smaller - changing a request in Chapter 08’s sense also changes when the HPA scales. It scales up as soon as the numbers say so. Down, it is slower on purpose: it keeps every recommendation of the last 300 seconds (the default stabilization window) and uses the highest - so after the load falls, one higher recommendation from a few minutes ago still holds the count up until it leaves the window. Without any metric (ScalingActive False) it changes nothing; when only some are missing, it may still scale up on the ones it has, but it does not scale down. And it does not replace a crashed pod: the ReplicaSet keeps the count the HPA last set, with or without metrics.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend
  minReplicas: 2
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

What a PodDisruptionBudget Guards

A PDB answers one question: may this pod be evicted now? It is asked by the Eviction API - and only by it.

The two meet in one place: pods that are not Ready - during a rollout, after an eviction, after a crash - count against the budget. A drain that starts in the middle of a rollout may find ALLOWED DISRUPTIONS 0, even though the rollout itself was never blocked.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: backend
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: backend

Reading a Budget

minAvailable counts the pods that must stay healthy; maxUnavailable counts the pods that may be gone. They are not two spellings of the same rule:

ALLOWED DISRUPTIONS is the number of evictions the budget would accept right now. It drops by one with each eviction and comes back only when the replacement is Ready - the lab watches it go from 1 to 0 and back.

The platform’s numbers:

minAvailable: 1 keeps one pod serving through any drain. It does not keep the capacity of a peak: with three pods it lets two go at once. Whether one pod can carry production’s load during a drain is a question about the load, not about the budget.

Two Replicas Are Not Two Nodes

Two replicas survive a crashed pod and make a rolling update seamless. They survive a lost node only if they run on two nodes. The scheduler spreads replicas when it can - it prefers different nodes, but does not insist. A topologySpreadConstraint (or pod anti-affinity) can insist; it needs a second node to spread to.

The platform’s honest limit: kind and the Hetzner cluster each have one schedulable worker. Every replica runs on it; a drain or a failure of that node takes everything on it. The budgets do their job - they keep a drain from emptying the node - but they cannot make room that does not exist. Making a drain invisible starts with a second worker, not with a setting.

Before a Drain

1. The nodes: which are schedulable (no taint, not cordoned), and how many.
2. The pods on the node, in every namespace - and for each: where its replacement would run.
3. The budgets of every namespace on that node: which show ALLOWED DISRUPTIONS 0, and why.
4. The simulation: kubectl drain --dry-run=server --ignore-daemonsets --delete-emptydir-data --timeout=... - it changes nothing, not even the cordon, and finds the budgets that block now. Each eviction is checked against unchanged budgets, so it cannot show what a real, sequential drain hits later - a second pod refused while the first one’s replacement is Pending. That is the placement and capacity check of steps 1-2.
5. The decision: go, or no-go with the reason. A real drain needs a timeout, someone watching, and kubectl uncordon afterwards - stopping a drain does not uncordon the node.

Guardrails That Stop It

• check-app-security.sh (workstation, CI) - refuses spec.replicas in Git for a Deployment with an HPA.
• PodDisruptionBudgets (cluster) - every app in every environment has one; the Postgres operator adds its own.
• The Eviction API (cluster) - a drain is refused, pod by pod, by the budgets; --disable-eviction and deleting pods bypass it, which is why the course never uses either.
• Flux (cluster) - a budget edited or deleted by hand comes back from Git at the next reconcile (Chapter 03).

Safe Workflow (Step-by-Step)

1. Read the HPA: conditions (ScalingActive), current and desired replicas, min and max.
2. Read every budget on the node - kubectl get pdb -A.
3. Read the placement: -o wide for the apps, and the schedulable nodes.
4. Simulate the drain with --dry-run=server and a --timeout.
5. Decide: a drain whose replacements have somewhere to go - or a planned outage, announced as one.
6. After a real drain: kubectl uncordon, then the pods Running and the budgets back to their allowed number.

Instructions for Your AI Agent

The rules this chapter adds:

1. Never put spec.replicas in Git for a Deployment an HPA scales. Enforced by: scripts/check-app-security.sh in pre-commit and CI.
2. Never drain a node without a plan: first a dry run that names the cluster with --context and the node explicitly - kubectl --context <cluster> drain <node> --dry-run=server --ignore-daemonsets --delete-emptydir-data --timeout=... - and the budgets of every namespace on that node (kubectl --context <cluster> get pdb -A), not only the one you work in; a real drain only with the owner’s yes, and uncordon after. Enforced by: instruction only.
3. Never lower or delete a PodDisruptionBudget, and never use --disable-eviction, to let a drain through. Report which budget blocks and why - only after reading its status (currentHealthy, desiredHealthy, expectedPods, the DisruptionAllowed condition): too few healthy pods, a budget as strict as the replica count, or a replacement with nowhere to run. Enforced by: the budgets are in Git (review), and Flux sets a hand edit back.
4. Do not promise node redundancy: check where the replicas run (-o wide) and how many schedulable nodes there are. Enforced by: instruction only.

The full list, chapter by chapter, is docs/agent-rules.md in your fork of the platform repository.

This builds on: Resource Management (Chapter 08) - the HPA measures usage against requests, and the quota must hold its maximum. This enables: Version Promotion (Chapter 10) - a rolling update of a promoted image relies on the replicas and budgets set here.