Chapter 15: Supply Chain Security (Advanced)

Incident Hook

An urgent fix is rebuilt from a developer workstation and pushed with a familiar tag. The deploy appears normal, but during incident triage the team cannot prove which workflow produced the binary. Dependency baseline, SBOM lineage, and signer identity are unclear. Rollback confidence drops because artifact trust is uncertain.

Observed Symptoms

What the team sees first:

• the workload is running, but artifact provenance is unclear
• tags look familiar while signer and SBOM evidence do not line up
• responders must investigate trust before they can trust rollback

The outage may be technical, but the recovery delay is a trust problem.

Confusion Phase

When an image “looks right,” teams want to keep moving. That is exactly when provenance work gets skipped.

The real question is:

• are we looking at the tested artifact
• or a fresh build that only resembles the intended release

Why This Chapter Exists

A successful CI build does not guarantee runtime trust. This chapter enforces a production rule: only verifiable artifacts may run.

The supply-chain baseline in this course is:

• immutable artifact identity (digest or immutable tag)
• SBOM generation
• image signing and attestation
• cluster-side verification before admission

Learning Objectives

By the end of this chapter, learners can:

• explain why “build once, promote many” is required for provenance
• generate and verify SBOM/signature evidence for one artifact
• run policy rollout in Audit -> Enforce phases in non-production
• document deny evidence and remediation path

What AI Would Propose (Brave Junior)

• “Rebuild locally and push now.”
• “Skip signing for this release only.”
• “Use mutable tags for faster retagging.”

Why this sounds reasonable:

• shortest time-to-deploy
• less CI friction under pressure

Why This Is Dangerous

• No cryptographic provenance at runtime.
• Rebuild variance breaks “tested artifact == deployed artifact”.
• Incident response becomes trust investigation instead of recovery.

Investigation

Start by proving artifact identity and signer evidence.

Safe investigation sequence:

1. verify digest, signature, and attestation for the running image
2. confirm the SBOM belongs to the same immutable artifact
3. compare runtime artifact identity with the tested promotion target
4. review policy reports to see whether the cluster would have denied an untrusted artifact

Containment

Containment restores trust before acceleration:

1. move back to the last known-good verified artifact if needed
2. re-run signing and attestation through the approved path
3. keep verification policy in Audit or Enforce according to evidence, not convenience
4. never “temporarily” bypass provenance checks to finish the release

Guardrails That Stop It

• Promote by artifact identity, not rebuild.
• SBOM is generated per release artifact.
• Signatures/attestations are verified before runtime admission.
• Verification policy starts in Audit, then moves to Enforce.

Prerequisites

• Kyverno policy engine installed and reconciling via Flux.
• Supply-chain policy pack templates available (see the investigation snapshots below).
• Rollout approach: start in Audit mode, collect deny evidence, then move to Enforce.

Investigation Snapshots

Here is the image verification policy template used in the SafeOps system to require signed images before rollout evidence is accepted.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images-example
spec:
  validationFailureAction: Audit
  background: false
  rules:
    - name: verify-ghcr-signatures
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/ldbl/*"
          mutateDigest: true
          attestors:
            - entries:
                - keyless:
                    # Replace with your CI issuer and subject constraints.
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/ldbl/*"

Here is the attestation policy template used for provenance and SBOM checks.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-sbom-attestation-example
spec:
  validationFailureAction: Audit
  background: false
  rules:
    - name: verify-spdx-attestation
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/ldbl/*"
          attestations:
            - type: "https://spdx.dev/Document"
              attestors:
                - entries:
                    - keyless:
                        issuer: "https://token.actions.githubusercontent.com"
                        subject: "https://github.com/ldbl/*"

System Context

This chapter extends GitOps promotion into cryptographic runtime trust.

It builds on:

• Chapter 04 immutable artifact identity
• Chapter 05 reviewed delivery paths
• Chapter 16, where broader admission policy guardrails enforce the same production boundary

Safe Workflow (Step-by-Step)

1. Pick immutable artifact reference.
2. Generate SBOM and sign/attest in CI-compatible flow.
3. Verify signature and attestation locally.
4. Apply verify policy in Audit mode in develop.
5. Observe reports/deny evidence and tune constraints.
6. Move selected rules to Enforce only after stable audit results.

Binary Outcome Drill (Must Demonstrate)

Target behavior in Enforce:

• unsigned/untrusted artifact: denied at admission
• signed + attested artifact (matching policy): admitted

Record both outcomes with policy report/event evidence as release criteria.

Lab Files

• lab.md
• runbook-supply-chain.md
• quiz.md

Done When

• learner proves artifact identity and provenance with command evidence
• learner demonstrates policy behavior in Audit and Enforce
• learner can handle unsigned/untrusted artifact deny without disabling guardrails

Handoff

Continue with Chapter 16 (Admission Policy Guardrails) for broader runtime policy enforcement beyond signatures.