Investigation

Treat admission logs and policy reports as the definitive record of cluster safety.

Safe investigation sequence:

1. Identify Violation: Use kubectl get policyreports or check cluster events for denied requests.
2. Trace Origin: Determine how the non-compliant workload reached the cluster (CI bypass, manual apply, GitOps drift).
3. Analyze Intent: Compare the rejected manifest with the intended security baseline.
4. Review Exceptions: Check if the failure was caused by a missing or misconfigured policy exception.

Containment

Containment is about re-establishing the cluster boundary without breaking existing legitimate workloads.

Containment steps:

1. Activate Audit Mode: Switch new or complex policies to Audit mode to gather evidence without blocking.
2. Remediate Upstream: Fix the root cause in the CI/CD pipeline or GitOps manifests.
3. Isolate Non-Compliant: Move existing non-compliant workloads to a restricted namespace for investigation.
4. Transition to Enforce: Shift policies back to Enforce mode only after the audit reports are clean.

The goal is “enforcement by default, exceptions by documentation.”

Pause and Predict: What automated guardrail would have prevented this incident entirely?