Chapter 15: Admission Policy Guardrails

Learning Objectives

By the end of this chapter, you will be able to:

• Deploy Kyverno policy packs in audit mode for safe rollout
• Graduate policies from audit to enforce using evidence from audit logs
• Configure break-glass exceptions with expiry and evidence requirements
• Evaluate policy coverage against the security baseline

Start with the video for the concept overview, then work through each lesson section.

Upstream guardrails (CI, PR reviews) are essential but can be bypassed. In this chapter, we implement a Cluster Gatekeeper using Kyverno. This is our final line of defense, ensuring that only workloads matching our production standards are admitted to the cluster.

1. The Problem: The “Policy Bypass”

A critical configuration error (like running as root or missing resource limits) slips through the CI/CD pipeline. Without a cluster-side check, the non-compliant workload runs in production, violating your security and stability invariants. The system’s safety depends entirely on every previous step being perfect.

2. The Concept: Cluster-Side Enforcement

We move from “hoping” the pipeline is correct to “enforcing” the state at the Kubernetes API boundary.

1. Admission Control: Every request to create or update a resource is intercepted by a policy engine.
2. Kyverno: A Kubernetes-native policy engine that uses standard YAML to define rules.
3. Fail-Closed: If a workload violates a critical policy (e.g., Security Context), the cluster rejects the request entirely.

3. The Code: The Policy Pack

Our sre/ repo includes standard policy packs for security, resources, and best practices. These policies are the definitive contract for what is allowed to run in our cluster.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-security-context-example
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: require-container-security-context
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Containers must run as non-root and disallow privilege escalation."
        pattern:
          spec:
            containers:
              - securityContext:
                  runAsNonRoot: true
                  allowPrivilegeEscalation: false

4. The Guardrail: Audit-First Rollout

We never block traffic blindly. Every new policy starts in Audit mode, where violations are recorded in a PolicyReport but not blocked. Only after the audit is clean do we move the policy to Enforce mode.

5. Verification: Did I Get It?

Check your current cluster compliance and test the boundary:

# Compliance reports for the scratch namespace
kubectl -n lab get policyreports
# Deploy a deliberately non-compliant pod there (latest tag, no limits, no securityContext)
kubectl -n lab run policy-test --image=nginx:latest
kubectl -n lab get policyreports -o wide

Expected Output: the pod runs (Kyverno is in Audit) and the report gains fail entries for it. Why lab and not develop? The environment namespaces enforce Pod Security restricted, which rejects this pod before Kyverno ever sees it — you would be observing PSS, not admission policy. lab only warns, so the policy engine is the one talking.

Detailed Lessons

• The Incident: The Policy Bypass
• Investigation & Containment
• Workflow & Strategy
• Lab & Completion
• Quiz