Core Track Guardrails-first chapter in core learning path.

Estimated Time

  • Reading: 20-25 min
  • Lab: 45-60 min
  • Quiz: 10-15 min

Prerequisites

Source Code References

  • .sops.yaml Members
  • sops-encrypt-secret.sh Members

Sign in to view source code.

What You Will Produce

One SOPS-encrypted secret in `flux/secrets/local/`, encrypted with your own key and decrypted by Flux into the `develop` namespace, plus a rehearsed first-30-minutes leak response.

Chapter 04: Secrets Management (SOPS + Age)

Learning Objectives

By the end of this chapter, you will be able to:

  • Encrypt secrets with SOPS and Age before committing to Git
  • Trace the Flux decryption flow from encrypted YAML to cluster Secret
  • Execute secret rotation after a leak incident
  • Explain why git revert is not remediation for an exposed secret

Start with the video for the concept overview, then work through each lesson section.

Plaintext secrets in Git are a production incident waiting to happen. In this chapter, we implement a secure GitOps secret workflow using SOPS and Age to ensure that sensitive data remains encrypted throughout its lifecycle.


1. The Problem: The Durable Leak

A teammate commits a plaintext API key to fix a failing deploy. Reverting the commit is not enough; the secret is already exposed in the Git history and CI logs. This creates a high-pressure situation requiring immediate rotation and auditing across the entire fan-out surface.

2. The Concept: Encrypted-at-Rest GitOps

We use SOPS (Mozilla Secrets Operations) and Age to keep our secrets safe. Secrets are encrypted on the developer’s machine before they are committed, and Flux decrypts them inside the cluster using a private key stored in a sops-age secret.

3. The Code: Secret Structure & Policies

Our sre/ repo defines exactly how and which files should be encrypted. The .sops.yaml policy ensures that only the relevant data fields are encrypted, keeping the YAML metadata visible for auditing.

SOPS encryption policy

creation_rules:
  # Cloudflare API token (cert-manager + external-dns)
  - path_regex: flux/secrets/cloudflare/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Auth secrets (Dex OIDC)
  - path_regex: flux/secrets/auth/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Observability secrets (shared)
  - path_regex: flux/secrets/observability/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Development environment secrets
  - path_regex: flux/secrets/develop/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Staging environment secrets
  - path_regex: flux/secrets/staging/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Production environment secrets
  - path_regex: flux/secrets/production/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Local (kind) profile - the learner's own key. Placeholder until
  # `scripts/sops-setup.sh --local` writes your public key here.
  - path_regex: flux/secrets/local/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1REPLACE_WITH_YOUR_PUBLIC_KEY_RUN_scripts_sops-setup.sh_--local

4. The Guardrail: Secret Encryption Helper

To make the “Safe Path” the easiest path, we use a helper script that automates the encryption process, preventing accidental plaintext commits.

Secret encryption helper

Show the secret encryption helper
#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

usage() {
    cat <<EOF
Usage: $0 ENVIRONMENT SECRET_NAME [NAMESPACE]

Create and encrypt a Kubernetes Secret with SOPS

ARGUMENTS:
    ENVIRONMENT     Target directory under flux/secrets/: develop, staging,
                    production, or local (the kind profile - encrypted with
                    your own key, see scripts/sops-setup.sh --local)
    SECRET_NAME     Name of the secret (e.g., backend-secrets)
    NAMESPACE       Namespace the Secret is created in (default: ENVIRONMENT,
                    or develop when ENVIRONMENT is local)

EXAMPLES:
    # Create encrypted secret for develop environment
    $0 develop backend-secrets

    # Create encrypted secret for production
    $0 production backend-secrets
    # Local kind profile: encrypted with your key, applied to develop
    $0 local lab-secret

WORKFLOW:
    1. Creates a plaintext secret template
    2. Opens it in \$EDITOR (or vi)
    3. After you save and quit, encrypts it with SOPS
    4. Saves encrypted version to flux/secrets/ENVIRONMENT/

NOTE:
    - Make sure .sops.yaml is configured with age public key
    - Make sure SOPS and age are installed
    - The plaintext file is automatically deleted after encryption

EOF
}

create_and_encrypt() {
    local env="$1"
    local secret_name="$2"
    local namespace="${3:-$env}"
    [[ "${env}" == "local" && -z "${3:-}" ]] && namespace="develop"
    local secrets_dir="${REPO_ROOT}/flux/secrets/${env}"
    local output_file="${secrets_dir}/${secret_name}.yaml"
    local temp_file="${secrets_dir}/${secret_name}.yaml.tmp"

    # Validate environment
    if [[ ! -d "${secrets_dir}" ]]; then
        echo "❌ Invalid environment: ${env}"
        echo "   Available: develop, staging, production, local"
        exit 1
    fi

    # Check if secret already exists
    if [[ -f "${output_file}" ]]; then
        echo "⚠️  Secret already exists: ${output_file}"
        read -p "   Edit existing secret with SOPS? (y/N): " -n 1 -r
        echo
        if [[ $REPLY =~ ^[Yy]$ ]]; then
            sops "${output_file}"
            echo "✅ Secret updated"
            exit 0
        else
            exit 1
        fi
    fi

    # Create template
    cat > "${temp_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${secret_name}
  namespace: ${namespace}
type: Opaque
stringData:
  # Add your secret keys here
  # Example:
  # database-url: "postgresql://user:pass@host:5432/db"
  # api-key: "your-api-key"
  # jwt-secret: "your-jwt-secret"

  # TODO: Replace with actual secret values
  example-key: "example-value"
EOF

    echo "📝 Created template: ${temp_file}"
    echo "   Opening in editor..."
    echo

    # Open in editor
    ${EDITOR:-vi} "${temp_file}"

    echo
    read -p "Encrypt this secret? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        echo "❌ Cancelled"
        rm "${temp_file}"
        exit 1
    fi

    # Encrypt with SOPS
    echo "🔐 Encrypting secret..."
    # The template ends in .tmp, so sops would guess "binary" and emit JSON
    # with the whole document in one blob - not a Secret manifest Flux can
    # apply. Force YAML in and out.
    sops --encrypt --input-type yaml --output-type yaml "${temp_file}" > "${output_file}"

    # Remove temp file
    rm "${temp_file}"

    echo "✅ Encrypted secret created: ${output_file}"
    echo
    echo "Next steps:"
    echo "  1. Add to kustomization: edit ${secrets_dir}/kustomization.yaml"
    echo "  2. Uncomment: # - ${secret_name}.yaml"
    echo "  3. Commit: git add ${output_file} && git commit -m 'Add ${secret_name} for ${env}'"
    echo "  4. Push: git push"
}

ENV="$1"
SECRET_NAME="$2"
NAMESPACE_ARG="${3:-}"

create_and_encrypt "${ENV}" "${SECRET_NAME}" "${NAMESPACE_ARG}"

5. Verification: Did I Get It?

Create an encrypted secret with your key (registered in Chapter 00) and verify that Flux applies it:

scripts/sops-encrypt-secret.sh local lab-secret      # encrypted for you, applied to develop
# add it to flux/secrets/local/kustomization.yaml, commit, push, then:
flux reconcile kustomization secrets-local --with-source
kubectl -n develop get secret lab-secret

The platform’s own secrets under flux/secrets/develop/ are encrypted for the platform key; you cannot decrypt them and you do not need to. Your lane is flux/secrets/local/.


Detailed Lessons

Hands-On Materials

Labs, quizzes, and runbooks — available to course members.

  • Lab: Encrypted Secret -> Flux Decrypt -> Cluster Apply Members
  • Quiz: Chapter 04 (Secrets Management with SOPS) Members

The Incident: The Durable Leak

Result: Response now includes rotation, audit, and cross-team coordination under pressure. Observed Symptoms What the team sees first: The deploy is unblocked, but the secret is visible in the PR diff. The same value may …

Investigation & Containment

Safe investigation sequence: Revoke immediately: Rotate the exposed credential at the source (e.g., GitHub, AWS, etc.). Identify dependencies: Identify downstream sessions, tokens, or integrations that depend on it. Map …

Safe Workflow & Tools

Generate a new key: age-keygen -o age-new.agekey. Update .sops.yaml: Replace the old public key with the new one. Re-encrypt: Run sops rotate --in-place &lt;file&gt; for every secret. Update Cluster: Delete and recreate …

Lab & Completion

Verification Checklist flux/secrets/local/lab-secret.yaml in Git is encrypted (ENC[...] values). secrets-local Kustomization is Ready in Flux. lab-secret exists in the develop namespace. No plaintext values appear in the …