Incident Hook

A critical vulnerability is discovered in an image that everyone thought was blocked. Because the earlier guardrails (CI, PR review) were bypassed or misconfigured, the untrusted image reached the cluster. Responders realize that without a cluster-side gatekeeper, there is no final line of defense against unauthorized mutations.

Result: The system’s safety depends entirely on upstream processes being perfect. A single mistake in the pipeline becomes a production vulnerability.

Observed Symptoms

What the team sees first:

• Workloads are running with configurations that violate company policy (e.g., running as root, missing labels).
• CI reports show green, yet the cluster state is non-compliant.
• Manual “quick fixes” bypass all established security controls.

The problem is the lack of a cluster-enforced source of truth.

Admission Policy Baseline

To ensure our production invariants are never violated, we implement a final line of defense at the Kubernetes API server boundary:

1. Mandatory Security Context: Every pod must run as non-root and have a read-only root filesystem.
2. Resource Limits: No pod is allowed to run without explicit CPU and memory limits.
3. No Privileged Containers: High-risk capabilities are blocked for all standard workloads.
4. Registry Restriction: Images can only be pulled from authorized container registries.

What AI Would Propose (Brave Junior):

• “Disable the admission controller temporarily to unblock the release.”
• “Apply ‘allow-all’ exceptions for development namespaces.”
• “Skip policy checks for images from ’trusted’ developers.”

Pause and Predict: Before reading the investigation, write down your top 3 hypotheses. What would you check first?