Workflow: Audit, Evidence, Enforce

Admission: The Last Question Before Storage

Every create or update reaches the API server the same way - from Flux, from kubectl, from an AI agent. Before the object is stored:

Authentication and RBAC decide whether this caller may do this at all (Chapter 01).
Mutating admission may change the object. The namespace’s LimitRange fills in missing requests and limits here (Chapter 08).
Validating admission may refuse it: Pod Security (Chapter 07), native ValidatingAdmissionPolicies (Chapter 13’s chaos-monkey-targets), and webhooks such as Kyverno’s. Each sees the object after every mutation.

So a Kyverno policy sees a pod with the LimitRange defaults already in it: a lab pod without resources passes the requests-and-limits guardrail in develop, because develop has a LimitRange. A Deployment does not get those defaults - LimitRange acts on pods - so it must set its own.

Each guardrail matches pods and the controllers that create them - Deployments, StatefulSets, DaemonSets, Jobs, CronJobs - and reads the pod template wherever that kind keeps it. A Deployment that breaks a guardrail is refused when it is applied, and Flux reports the reason; without that, the Deployment would be stored and its ReplicaSet would fail to create pods, with the reason buried in its events.

The Policy Language: CEL

The guardrails are Kyverno ValidatingPolicies: each check is a CEL expression - the Common Expression Language the API server itself uses for native ValidatingAdmissionPolicies (Chapter 01) - and Kyverno adds what the native policies do not have: reports on what already runs, exceptions, functions such as image(). One guardrail, shortened:

apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
spec:
  validationActions: [Deny]                 # refuse (Audit: only record)
  matchConstraints:                          # which requests it sees
    resourceRules:
      - {apiGroups: [""], apiVersions: [v1], operations: [CREATE, UPDATE], resources: [pods, pods/ephemeralcontainers]}
      - {apiGroups: [apps], apiVersions: [v1], operations: [CREATE, UPDATE], resources: [deployments, statefulsets, daemonsets]}
      - {apiGroups: [batch], apiVersions: [v1], operations: [CREATE, UPDATE], resources: [jobs, cronjobs]}
  variables:
    - name: podSpec                          # the pod template, wherever this kind keeps it
      expression: >-
        object.kind == 'Pod' ? object.spec
        : object.kind == 'CronJob' ? object.spec.jobTemplate.spec.template.spec
        : object.spec.template.spec
    - name: latest                           # every image whose tag is "latest" and that has no digest
      expression: >-
        (variables.podSpec.containers + variables.podSpec.?initContainers.orValue([]) + variables.podSpec.?ephemeralContainers.orValue([]))
        .map(c, c.image).filter(i, !image(i).containsDigest() && image(i).tag() == 'latest')
  validations:
    - expression: size(variables.latest) == 0
      messageExpression: "'Image tag latest (or no tag) is not allowed ...: ' + variables.latest.join(', ')"

Read it top to bottom: which requests, what to compute, what must be true. ?. and orValue([]) read a field that may be missing - a pod without init containers - without an error. The message names what it found, so the person refused does not have to guess.

Kyverno’s older policy kind, ClusterPolicy with YAML patterns, still works but is deprecated: Kyverno 1.19 warns on every request for one. This platform moved the five guardrails to ValidatingPolicies - the migration Chapter 20’s check of deprecated APIs points at. Kyverno can also generate the controller variants of a pod policy by itself (autogen), but only for a policy that matches pods alone; these also match pods/ephemeralcontainers - what kubectl debug writes - so the controllers are listed, and visible.

Audit and Enforce, Per Namespace

A violation is either recorded - the request goes through, and a PolicyReport entry says fail - or refused: the API server answers with the policy’s message, and nothing is stored. A ValidatingPolicy has one action for every namespace it matches (validationActions: [Deny] or [Audit]), so each guardrail is two policies in one file, identical except for these lines:

# <name>-enforce                              # <name>-audit
validationActions: [Deny]                     validationActions: [Audit]
failurePolicy: Fail                           failurePolicy: Ignore
namespaceSelector:                            namespaceSelector:
  matchExpressions:                             matchExpressions:
    - key: kubernetes.io/metadata.name            - key: kubernetes.io/metadata.name
      operator: In                                  operator: NotIn
      values: [develop, staging, production]        values: [develop, staging, production]

The enforce copy refuses in the application namespaces - and refuses while Kyverno cannot answer (failurePolicy: Fail). The audit copy records everywhere else, and lets the request through when Kyverno is down: a report is not worth blocking the platform for. Two copies of one rule can drift apart - a fix made in one only - so the test (Tested Before Merge, below) compares them and fails when they differ in anything but these lines.

Two limits on what Enforce refuses:

Only what the webhooks see. kube-system, kyverno and flux-system are outside Kyverno’s webhooks (When the Engine Is Down, below): no copy refuses anything there; their violations only reach the PolicyReports of background scans.
New violations, not old ones. A request that creates a violating object is refused, and so is one that changes its spec - a new image, an added debug container. An update that leaves the spec as it was - a label, an annotation, a finalizer removed - is let through (a metadataOnly variable in each policy): a pod that already broke the rule - admitted before the rule, or under an exception that has since expired - would otherwise be stuck, unable even to lose a finalizer and finish terminating. So a running workload is refused when its next pod is created - a restart that makes a new pod, a rollout, a node replaced - and the background scan keeps reporting it until then.

The Five Guardrails

Guardrail (two policies: `-enforce`, `-audit`)	Refuses (in the application namespaces)	Why it is not left to the review
`disallow-latest-tag`	an image tagged `latest`, or with no tag - which pulls `latest` - in any container, init or ephemeral (`kubectl debug`) included	`latest` is the tag that moves by design: the same manifest can run a different image tomorrow. The rule refuses only that riskiest case - any other tag passes, though most registries let a tag be moved too; only a digest names one image (the rollback of Chapter 10 uses the image tags of its own registry)
`require-trusted-registries`	an image not from the platform’s registry, CloudNativePG, or the three lab images (`busybox`, `postgres`, `curlimages/curl`) - in any container, init or ephemeral included	an unknown registry is unknown code; Docker Hub is allowed image by image, not as `docker.io/*`, which would be any account
`disallow-privileged-containers`	`privileged: true` in a container, an init container or an ephemeral one (`kubectl debug --profile=sysadmin` adds exactly that)	one escape from it is an escape from the node
`require-security-context`	a container that may run as root, or may gain privileges	least privilege for the process; `runAsNonRoot` may be set for the pod or per container
`require-requests-limits`	a container or an init container without CPU and memory requests and limits (ephemeral containers cannot have them)	scheduling and the quota of Chapter 08 depend on them

disallow-privileged-containers and require-security-context overlap with Pod Security restricted, which already refuses those settings in the application namespaces; Kyverno repeats them for the report in namespaces Pod Security does not restrict. The other three are rules Pod Security does not have: it checks neither image tags nor registries, nor requests and limits - a namespace with only restricted admits a container with no resources at all.

disallow-latest-tag

Show disallow-latest-tag.yaml

# A mutable tag ("latest", or no tag at all - the runtime pulls "latest" then) means the same manifest
# can run a different image tomorrow: no artifact identity, no reliable rollback (Chapter 10).
# image() parses a reference the way the runtime does: `curlimages/curl` has the tag "latest", and an
# image pinned by digest passes whatever its tag says. Init and ephemeral containers count too.
# Any other tag passes, though most registries let a tag be moved: only a digest names one image.
#
# Two copies, identical but for the name, the action, the failure policy and the namespace operator
# (tests/kyverno-policies.test.sh fails when they drift apart):
#   disallow-latest-tag-enforce  develop, staging, production - Deny, failurePolicy Fail
#   disallow-latest-tag-audit    every other namespace         - Audit (a PolicyReport), failurePolicy Ignore
# Exceptions: flux/infrastructure/policy/exceptions/, through a pull request.
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-latest-tag-enforce
  annotations:
    policies.kyverno.io/title: Disallow Latest Tag (enforce)
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
spec:
  validationActions: [Deny]
  failurePolicy: Fail
  evaluation:
    background:
      enabled: true
  matchConstraints:
    # Pods - and the controllers that create them, listed here instead of Kyverno's autogen: autogen
    # only works for a policy that matches pods alone, and kubectl debug's pods/ephemeralcontainers
    # must be matched too. A bad Deployment is refused at apply, not later when its pods are created.
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods, pods/ephemeralcontainers]
      - apiGroups: [apps]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [deployments, statefulsets, daemonsets]
      - apiGroups: [batch]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [jobs, cronjobs]
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [develop, staging, production]
  variables:
    # The pod spec, wherever this kind keeps it.
    - name: podSpec
      expression: >-
        object.kind == 'Pod' ? object.spec
        : object.kind == 'CronJob' ? object.spec.jobTemplate.spec.template.spec
        : object.spec.template.spec
    # An UPDATE that leaves the spec as it was - a label, an annotation, a finalizer removed. It is let
    # through: ClusterPolicy did the same (allowExistingViolations), and refusing it would leave a pod
    # that already breaks the rule - one admitted before the rule, or under an exception that has since
    # expired - unable to get a label or lose a finalizer, stuck in Terminating. A changed spec, an
    # added ephemeral container and every CREATE are checked; the background scan still reports the
    # existing violation (it has no request, so this is false there).
    - name: metadataOnly
      expression: >-
        request.?operation.orValue('') == 'UPDATE' && oldObject != null && has(oldObject.spec)
        && object.spec == oldObject.spec
    # Every image whose tag is "latest" (written or implied) and that has no digest.
    - name: latest
      expression: >-
        (variables.podSpec.containers + variables.podSpec.?initContainers.orValue([]) + variables.podSpec.?ephemeralContainers.orValue([]))
        .map(c, c.image).filter(i, !image(i).containsDigest() && image(i).tag() == 'latest')
  validations:
    - expression: variables.metadataOnly || size(variables.latest) == 0
      messageExpression: >-
        'Image tag latest (or no tag) is not allowed - use an immutable tag or a digest: '
        + variables.latest.join(', ')
---
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-latest-tag-audit
  annotations:
    policies.kyverno.io/title: Disallow Latest Tag (audit)
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
spec:
  validationActions: [Audit]
  failurePolicy: Ignore
  evaluation:
    background:
      enabled: true
  matchConstraints:
    # Pods - and the controllers that create them, listed here instead of Kyverno's autogen: autogen
    # only works for a policy that matches pods alone, and kubectl debug's pods/ephemeralcontainers
    # must be matched too. A bad Deployment is refused at apply, not later when its pods are created.
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods, pods/ephemeralcontainers]
      - apiGroups: [apps]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [deployments, statefulsets, daemonsets]
      - apiGroups: [batch]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [jobs, cronjobs]
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [develop, staging, production]
  variables:
    # The pod spec, wherever this kind keeps it.
    - name: podSpec
      expression: >-
        object.kind == 'Pod' ? object.spec
        : object.kind == 'CronJob' ? object.spec.jobTemplate.spec.template.spec
        : object.spec.template.spec
    # An UPDATE that leaves the spec as it was - a label, an annotation, a finalizer removed. It is let
    # through: ClusterPolicy did the same (allowExistingViolations), and refusing it would leave a pod
    # that already breaks the rule - one admitted before the rule, or under an exception that has since
    # expired - unable to get a label or lose a finalizer, stuck in Terminating. A changed spec, an
    # added ephemeral container and every CREATE are checked; the background scan still reports the
    # existing violation (it has no request, so this is false there).
    - name: metadataOnly
      expression: >-
        request.?operation.orValue('') == 'UPDATE' && oldObject != null && has(oldObject.spec)
        && object.spec == oldObject.spec
    # Every image whose tag is "latest" (written or implied) and that has no digest.
    - name: latest
      expression: >-
        (variables.podSpec.containers + variables.podSpec.?initContainers.orValue([]) + variables.podSpec.?ephemeralContainers.orValue([]))
        .map(c, c.image).filter(i, !image(i).containsDigest() && image(i).tag() == 'latest')
  validations:
    - expression: variables.metadataOnly || size(variables.latest) == 0
      messageExpression: >-
        'Image tag latest (or no tag) is not allowed - use an immutable tag or a digest: '
        + variables.latest.join(', ')

The two image guardrails do not compare the image string; they parse it with Kyverno’s CEL function image() - registry, repository, tag, digest - the way the container runtime reads it. curlimages/curl has no tag in the manifest, but the runtime pulls curlimages/curl:latest, and image('curlimages/curl').tag() says latest. busybox:1.38 and docker.io/library/busybox:1.38 are written differently and are the same image: both parse to registry index.docker.io, repository library/busybox. A rule on the written string would miss the first and refuse the second.

require-trusted-registries

Show require-trusted-registries.yaml

# Only images from known sources may run: the platform's own registry, the CloudNativePG images and
# the three Docker Hub images the labs use. Everything else needs an exception through a pull request.
# image() parses the reference the way the runtime does, so `busybox:1.38` and
# `docker.io/library/busybox:1.38` are the same image (registry index.docker.io, repository
# library/busybox). Docker Hub official images are listed one by one, not as all of docker.io: that
# would admit any Docker Hub account. Init and ephemeral containers (kubectl debug) are checked too.
#
# Two copies, identical but for the name, the action, the failure policy and the namespace operator
# (tests/kyverno-policies.test.sh fails when they drift apart):
#   require-trusted-registries-enforce  develop, staging, production - Deny, failurePolicy Fail
#   require-trusted-registries-audit    every other namespace         - Audit (a PolicyReport), failurePolicy Ignore
# Exceptions: flux/infrastructure/policy/exceptions/, through a pull request.
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-trusted-registries-enforce
  annotations:
    policies.kyverno.io/title: Require Trusted Container Registries (enforce)
    policies.kyverno.io/category: Supply Chain Security
    policies.kyverno.io/severity: high
spec:
  validationActions: [Deny]
  failurePolicy: Fail
  evaluation:
    background:
      enabled: true
  matchConstraints:
    # Pods - and the controllers that create them, listed here instead of Kyverno's autogen: autogen
    # only works for a policy that matches pods alone, and kubectl debug's pods/ephemeralcontainers
    # must be matched too. A bad Deployment is refused at apply, not later when its pods are created.
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods, pods/ephemeralcontainers]
      - apiGroups: [apps]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [deployments, statefulsets, daemonsets]
      - apiGroups: [batch]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [jobs, cronjobs]
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [develop, staging, production]
  variables:
    # The pod spec, wherever this kind keeps it.
    - name: podSpec
      expression: >-
        object.kind == 'Pod' ? object.spec
        : object.kind == 'CronJob' ? object.spec.jobTemplate.spec.template.spec
        : object.spec.template.spec
    # An UPDATE that leaves the spec as it was - a label, an annotation, a finalizer removed. It is let
    # through: ClusterPolicy did the same (allowExistingViolations), and refusing it would leave a pod
    # that already breaks the rule - one admitted before the rule, or under an exception that has since
    # expired - unable to get a label or lose a finalizer, stuck in Terminating. A changed spec, an
    # added ephemeral container and every CREATE are checked; the background scan still reports the
    # existing violation (it has no request, so this is false there).
    - name: metadataOnly
      expression: >-
        request.?operation.orValue('') == 'UPDATE' && oldObject != null && has(oldObject.spec)
        && object.spec == oldObject.spec
    # Every image not from a trusted source. A reference is compared as registry/repository, without
    # the tag or digest; ${image_registry} and ghcr.io/cloudnative-pg are trusted with everything below
    # them, the Docker Hub images by exact repository.
    - name: untrusted
      expression: >-
        (variables.podSpec.containers + variables.podSpec.?initContainers.orValue([]) + variables.podSpec.?ephemeralContainers.orValue([]))
        .map(c, c.image).filter(i,
          !(image(i).registry() + '/' + image(i).repository()).startsWith('${image_registry}/')
          && !(image(i).registry() + '/' + image(i).repository()).startsWith('ghcr.io/cloudnative-pg/')
          && !(image(i).registry() == 'index.docker.io'
               && image(i).repository() in ['library/busybox', 'library/postgres', 'curlimages/curl']))
  validations:
    - expression: variables.metadataOnly || size(variables.untrusted) == 0
      messageExpression: >-
        'Images must come from ${image_registry}/*, ghcr.io/cloudnative-pg/* or be one of the lab images
        (docker.io busybox, postgres, curlimages/curl). Not trusted here: ' + variables.untrusted.join(', ')
---
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: require-trusted-registries-audit
  annotations:
    policies.kyverno.io/title: Require Trusted Container Registries (audit)
    policies.kyverno.io/category: Supply Chain Security
    policies.kyverno.io/severity: high
spec:
  validationActions: [Audit]
  failurePolicy: Ignore
  evaluation:
    background:
      enabled: true
  matchConstraints:
    # Pods - and the controllers that create them, listed here instead of Kyverno's autogen: autogen
    # only works for a policy that matches pods alone, and kubectl debug's pods/ephemeralcontainers
    # must be matched too. A bad Deployment is refused at apply, not later when its pods are created.
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [pods, pods/ephemeralcontainers]
      - apiGroups: [apps]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [deployments, statefulsets, daemonsets]
      - apiGroups: [batch]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [jobs, cronjobs]
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [develop, staging, production]
  variables:
    # The pod spec, wherever this kind keeps it.
    - name: podSpec
      expression: >-
        object.kind == 'Pod' ? object.spec
        : object.kind == 'CronJob' ? object.spec.jobTemplate.spec.template.spec
        : object.spec.template.spec
    # An UPDATE that leaves the spec as it was - a label, an annotation, a finalizer removed. It is let
    # through: ClusterPolicy did the same (allowExistingViolations), and refusing it would leave a pod
    # that already breaks the rule - one admitted before the rule, or under an exception that has since
    # expired - unable to get a label or lose a finalizer, stuck in Terminating. A changed spec, an
    # added ephemeral container and every CREATE are checked; the background scan still reports the
    # existing violation (it has no request, so this is false there).
    - name: metadataOnly
      expression: >-
        request.?operation.orValue('') == 'UPDATE' && oldObject != null && has(oldObject.spec)
        && object.spec == oldObject.spec
    # Every image not from a trusted source. A reference is compared as registry/repository, without
    # the tag or digest; ${image_registry} and ghcr.io/cloudnative-pg are trusted with everything below
    # them, the Docker Hub images by exact repository.
    - name: untrusted
      expression: >-
        (variables.podSpec.containers + variables.podSpec.?initContainers.orValue([]) + variables.podSpec.?ephemeralContainers.orValue([]))
        .map(c, c.image).filter(i,
          !(image(i).registry() + '/' + image(i).repository()).startsWith('${image_registry}/')
          && !(image(i).registry() + '/' + image(i).repository()).startsWith('ghcr.io/cloudnative-pg/')
          && !(image(i).registry() == 'index.docker.io'
               && image(i).repository() in ['library/busybox', 'library/postgres', 'curlimages/curl']))
  validations:
    - expression: variables.metadataOnly || size(variables.untrusted) == 0
      messageExpression: >-
        'Images must come from ${image_registry}/*, ghcr.io/cloudnative-pg/* or be one of the lab images
        (docker.io busybox, postgres, curlimages/curl). Not trusted here: ' + variables.untrusted.join(', ')

Rollout: Audit, Evidence, Enforce

A new rule - or an existing rule in a new namespace - goes through four steps, each in a pull request:

Audit everywhere. The rule records; nothing is refused.
Read the evidence. The PolicyReports of the namespaces where it will enforce show no failures - and the workloads that are not there right now, checked by hand: lab pods, Jobs, a restore’s pods.
Fix the workloads, not the rule. A failure is fixed in the workload’s manifest; the rule stays as it is.
Enforce where it is clean. Those namespaces added to the guardrail’s namespace list - in both copies, the enforce copy’s In and the audit copy’s NotIn - with the evidence in the pull request. Everywhere else stays in Audit, and its reports stay the backlog.

Audit is the evidence for Enforce, not a place to stay: each rule in Audit has an owner and a reason it is not enforced yet. On this platform the platform namespaces stay in Audit because their charts break several rules - fixing those charts is its own work, not a reason to keep the application namespaces unprotected.

Tested Before Merge

A policy is code, and a typo in it can admit everything or refuse everything. tests/kyverno-policies.test.sh runs kyverno test - the Kyverno engine without a cluster - against a set of objects with a known verdict each: a compliant pod, one tagged latest, one with no tag, one from an unknown registry, one with a privileged init container, a Deployment and a CronJob, the same bad pod in a namespace that only audits, one covered by an exception. It runs in pre-commit and in CI (Chapter 05): a change that turns a fail into a pass fails the pull request before it can reach the cluster. It also fails when:

a policy - each copy - has no expected pass or no expected fail: a policy tested only one way could be broken the other way, refusing everything or nothing;
the two copies of a guardrail differ in anything but name, title, action, failure policy and namespace operator - a fix made in one copy only.

Exceptions: Narrow, Owned, Temporary - and Only From Git

Some workload will need to break a rule for a while: a vendor agent from its own registry, until it is mirrored. Switching the guardrail to Audit, or narrowing its namespaces, would let everything else through as well. A Kyverno PolicyException lets one named workload past one policy:

apiVersion: policies.kyverno.io/v1
kind: PolicyException
metadata:
  name: develop-vendor-agent
  namespace: policy-exceptions          # the only namespace Kyverno reads exceptions from
  annotations:
    safeops.io/owner: platform-team
    safeops.io/reason: "https://github.com/<you>/sre/issues/42 - until the agent is mirrored"
spec:
  expiresAt: "<YYYY-MM-DD>T00:00:00Z"   # at most 90 days ahead - the lab computes it (B7)
  policyRefs:
    - name: require-trusted-registries-enforce   # the copy that judges develop
      kind: ValidatingPolicy
  matchConditions:
    - name: one-workload
      expression: "object.metadata.namespace == 'develop' && object.metadata.name.startsWith('vendor-agent')"

What it matches is a CEL condition, and CEL can say anything - every pod in a namespace, every pod with a label. So the platform allows one form only: one namespace, and a name - object.metadata.name == 'vendor-agent' for one pod, or .startsWith('vendor-agent') for a Deployment and its pods, which are named vendor-agent-<hash>-<id>. A prefix also matches vendor-agent-2; for a single pod, the exact name is the narrower choice.

Three guards keep it narrow:

Where Kyverno reads it. Only the policy-exceptions namespace. An exception anywhere else does nothing.
Who may write it. A native ValidatingAdmissionPolicy, policy-exceptions-only-from-git, lets only Flux’s kustomize-controller create or change a PolicyException. A kubectl apply - by a person or an AI agent, cluster-admin included - is refused. An exception is then always a pull request: reviewed, recorded, and reverted the same way. The guard is a native policy on purpose: it guards the way around Kyverno, so it must not depend on Kyverno.
What it may say. scripts/check-policy-exceptions.sh (pre-commit and CI) refuses an exception without an owner, a reason and an expiresAt at most 90 days ahead; one that names more than one policy, or a policy that is not one of the guardrails - a typo would except nothing and look as if it did, and Chapter 17’s signature check takes no exceptions at all; and a matchConditions entry in any form but the one above.
When it ends. Kyverno stops applying an exception at its expiresAt: from then on the workload is judged like any other - a new pod of it is refused. The file is still in Git, though, and says the exception is in force. So the date is also detected: an exception past its date fails every pull request - CI runs the check each time - and a daily run of the same check on main (policy-exceptions-expiry.yml) fails even when nobody opens one, until a pull request removes or renews it - which is why its owner is named.

Only Flux writes PolicyExceptions

Show only-from-git.yaml

# A PolicyException switches a guardrail off for what it names - so who may write one matters as
# much as the guardrail. This policy lets only Flux (kustomize-controller, applying Git) create or
# change PolicyExceptions; a kubectl apply by a person or an AI agent - even with cluster-admin - is
# refused. Deleting one is not guarded: it can only make the platform stricter, and Flux re-applies
# it while it is still in Git.
# Break-glass (Flux itself broken): the admin kubeconfig deletes this binding, writes the exception,
# and the same change goes into Git right after - Flux then restores the binding.
# Native ValidatingAdmissionPolicy, Deny from the start - it guards the exception path itself, and it
# must not depend on Kyverno, the engine it is guarding.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: policy-exceptions-only-from-git
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      # policies.kyverno.io: exceptions to the CEL policies (the ones in use); kyverno.io: the older
      # kind, for ClusterPolicies - guarded too, so that it cannot be a way around.
      - apiGroups: ["kyverno.io", "policies.kyverno.io"]
        apiVersions: ["*"]
        operations: ["CREATE", "UPDATE"]
        resources: ["policyexceptions"]
  validations:
    - expression: "request.userInfo.username == 'system:serviceaccount:flux-system:kustomize-controller'"
      messageExpression: >-
        'PolicyExceptions come only from Git (flux/infrastructure/policy/exceptions, through a pull request); '
        + request.userInfo.username + ' may not write them'
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: policy-exceptions-only-from-git
spec:
  policyName: policy-exceptions-only-from-git
  validationActions: [Deny]

Break-glass - Flux itself broken, an exception needed now - is the owner with the admin kubeconfig: delete the binding of that policy, write the exception, and put the same change into Git right after; Flux restores the binding. That is an escalation, not a workaround (Chapter 15). Nobody else can take this path: the binding is cluster-wide, and the everyday OIDC roles may write only in develop (Chapter 01) - deleting it needs the break-glass kubeconfig.

When the Engine Is Down

How Kyverno goes down decides what happens - measured on kind:

Kyverno…	Its webhooks	A new pod in `develop`
crashes (a pod killed, a node lost) and no replica answers	stay registered, `failurePolicy: Fail`	refused - fail closed
admission controller stopped on purpose (scaled to zero)	deleted by Kyverno on its way out - its design for a clean uninstall	admitted, unchecked by Kyverno - no Kyverno rule runs at admission until it starts and registers them again

In the second row two things keep working: Kyverno’s background scans, which still write PolicyReports for what was admitted, and the native ValidatingAdmissionPolicies below, which do not depend on Kyverno at all. Uninstalling Kyverno removes the scans as well.

Fail closed is the right default for a guardrail - a rule that is skipped whenever its engine is slow is not a rule - and it makes Kyverno a dependency of every deployment. The second row makes stopping the admission controller a way around every Kyverno rule. So the platform designs both:

Two replicas, one per node, and a PodDisruptionBudget keep one answering through a node drain. One per node is not a detail: the Hetzner cluster has one worker, and with both replicas on it a reboot took both down - the fail-closed webhook refused every new application pod, and the ReplicaSets, backing off between retries, kept production down 18 minutes for a 5-minute reboot (2026-10-12). So the admission controller now tolerates the control plane’s taint - it alone, not every workload - and a topology spread puts one replica per node. On kind the two replicas land on the two nodes, after a rollout and after a pod is deleted; that the reboot then costs production only the reboot is for the next Hetzner cycle to show.
Stopping Kyverno takes break-glass or a reviewed pull request. The everyday OIDC roles have no write access in the kyverno namespace (Chapter 01). Scaling or removing it otherwise is a change to Kyverno’s HelmRelease in Git - reviewed like the policies, because Flux would carry it out.
kube-system, kyverno and flux-system are outside the webhooks. The first two keep the cluster and Kyverno itself able to start. flux-system keeps the repair path open: a broken Kyverno is fixed by Flux applying Git, and Flux must not wait for Kyverno to do it. Their pods are still scanned in the background and reported.
The guards that protect the access model are native policies, evaluated inside the API server: oidc-flux-operator-fields, oidc-flux-objects-from-git, chaos-monkey-targets and policy-exceptions-only-from-git work while Kyverno is down.

Kyverno or a Native Policy?

	Native ValidatingAdmissionPolicy	Kyverno
Runs	inside the API server	as webhooks - another service that must answer
Language	CEL expressions	CEL expressions (ValidatingPolicy) - the same language, plus Kyverno’s functions
Reports on what already runs	no - it only judges requests	PolicyReports from background scans
Exceptions	by changing the binding or the policy	PolicyExceptions, scoped and reviewable
Parses image references	no - you parse the string yourself	yes - `image()`
Used here for	guards of the access model, which must work while Kyverno is down	the workload guardrails, which need reports and exceptions

Instructions for Your AI Agent

The rules this chapter adds:

When admission refuses a change, fix the manifest - never weaken the policy, switch it to Audit, or narrow its namespaces to get one workload through. Enforced by: review of the policy diff; tests/kyverno-policies.test.sh (pre-commit and CI) fails when a policy’s verdicts change, or its two copies drift apart.
An exception names one policy, one namespace and one workload - by its exact name, or for a Deployment by a name prefix that no other workload in the namespace starts with - in the fixed matchConditions form, with an owner (safeops.io/owner), a reason (safeops.io/reason) and an expiresAt at most 90 days ahead - and goes in through a pull request. Enforced by: scripts/check-policy-exceptions.sh (pre-commit and CI); the ValidatingAdmissionPolicy policy-exceptions-only-from-git refuses any writer but Flux.
Never kubectl apply a policy, a mode change or an exception by hand, and never delete the policy-exceptions-only-from-git binding - that is the owner’s break-glass, when Flux itself is broken. Enforced by: the ValidatingAdmissionPolicy for exceptions; Flux reverts hand changes to the policies (Chapter 03).
Before proposing Enforce for a rule or a namespace, show the evidence: that namespace’s PolicyReports with no failures, and the occasional workloads (lab pods, Jobs) checked too. Enforced by: review of the pull request.

This builds on: 24/7 Production SRE (Chapter 15) and the core track - the rules turn the runtime baseline of Chapters 07 to 10 into checks the cluster itself makes. This enables: Supply Chain Security (Chapter 17) - the same engine verifies who built an image, and its policies go through the same Audit, evidence, Enforce.