Chapter 03: Secrets Management (SOPS)

Incident Hook

A teammate commits a plaintext API key to fix a failing deploy quickly. The key is exposed in Git history, CI logs, and local clones. The rollback is not enough because the secret is already leaked. Response now includes rotation, audit, and cross-team coordination under pressure.

Observed Symptoms

What the team sees first:

• the deploy is unblocked, but the secret is visible in the diff
• the same value may now exist in PR tooling, CI logs, and local clones
• responders cannot tell immediately how many copies of the credential exist

The visible file change is only the smallest part of the incident.

Confusion Phase

The first instinct is usually to revert the commit and move on. That is where teams lose time.

The real questions are:

• is the credential still valid
• where else was it copied while exposed
• has the replacement path already created a second unsafe secret

Why This Chapter Exists

Plaintext secrets in Git are a production incident waiting to happen. This chapter establishes one safe path:

• secrets are encrypted before commit
• Flux decrypts in-cluster with sops-age
• key material is never committed

What AI Would Propose (Brave Junior)

• “Create Kubernetes Secret YAML and push it fast.”
• “We can encrypt later.”

Why this sounds reasonable:

• fastest path to unblock deployment
• appears reversible via git revert

Why This Is Dangerous

• Git history is durable; reverting does not un-leak the value.
• Secret fan-out is unknown (clones, caches, logs, screenshots).
• Blast radius includes external integrations using that credential.

Investigation

Treat a leaked secret as a trust incident, not a formatting mistake.

Safe investigation sequence:

1. revoke or rotate the exposed credential first
2. identify downstream sessions, tokens, or integrations that depend on it
3. map the leak surface across Git history, CI output, chat, screenshots, and local clones
4. confirm the replacement credential moved only through the encrypted path

The goal is to restore trust, not just revert a bad file.

Containment

Containment is simple but strict:

1. invalidate the leaked value
2. distribute the replacement via SOPS only
3. verify Flux can decrypt and apply it
4. record follow-up hardening work before closing the incident

Guardrails That Stop It

• No plaintext secrets under flux/secrets/**.
• sops-age secret must exist in flux-system before relying on encrypted manifests.
• Only encrypted files are allowed in PRs.
• Secret rotation plan is mandatory after any exposure.
• Local no-secrets pre-commit hook blocks common sensitive files before commit.
• Local flux-kustomize-validate pre-commit hook catches broken Flux Kustomize wiring before commit.

Leak Response Mini-Runbook (First 30 Minutes)

1. Rotate/revoke exposed credential immediately.
2. Invalidate active sessions/tokens that depend on the secret.
3. Identify leak surface (Git history, CI logs, chat snippets, local copies).
4. Confirm replacement secret is distributed via encrypted path only.
5. Open incident record and notify affected service/security owners.
6. Create follow-up tasks: audit access logs, remove stale credentials, harden detection.

Investigation Snapshots

Here is the SOPS policy used in the SafeOps system to decide how secrets are encrypted before they ever enter Git history.

creation_rules:
  # Cloudflare API token (cert-manager + external-dns)
  - path_regex: flux/secrets/cloudflare/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Observability secrets (shared)
  - path_regex: flux/secrets/observability/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Development environment secrets
  - path_regex: flux/secrets/develop/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Staging environment secrets
  - path_regex: flux/secrets/staging/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Production environment secrets
  - path_regex: flux/secrets/production/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

Here is the helper used on the safe path to create and encrypt a secret without committing plaintext.

#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

usage() {
    cat <<EOF
Usage: $0 ENVIRONMENT SECRET_NAME

Create and encrypt a Kubernetes Secret with SOPS

ARGUMENTS:
    ENVIRONMENT     Target environment (develop, staging, production)
    SECRET_NAME     Name of the secret (e.g., backend-secrets)

EXAMPLES:
    # Create encrypted secret for develop environment
    $0 develop backend-secrets

    # Create encrypted secret for production
    $0 production backend-secrets

WORKFLOW:
    1. Creates a plaintext secret template
    2. Opens it in \$EDITOR (or vi)
    3. After you save and quit, encrypts it with SOPS
    4. Saves encrypted version to flux/secrets/ENVIRONMENT/

NOTE:
    - Make sure .sops.yaml is configured with age public key
    - Make sure SOPS and age are installed
    - The plaintext file is automatically deleted after encryption

EOF
}

create_and_encrypt() {
    local env="$1"
    local secret_name="$2"
    local secrets_dir="${REPO_ROOT}/flux/secrets/${env}"
    local output_file="${secrets_dir}/${secret_name}.yaml"
    local temp_file="${secrets_dir}/${secret_name}.yaml.tmp"

    # Validate environment
    if [[ ! -d "${secrets_dir}" ]]; then
        echo "❌ Invalid environment: ${env}"
        echo "   Available: develop, staging, production"
        exit 1
    fi

    # Check if secret already exists
    if [[ -f "${output_file}" ]]; then
        echo "⚠️  Secret already exists: ${output_file}"
        read -p "   Edit existing secret with SOPS? (y/N): " -n 1 -r
        echo
        if [[ $REPLY =~ ^[Yy]$ ]]; then
            sops "${output_file}"
            echo "✅ Secret updated"
            exit 0
        else
            exit 1
        fi
    fi

    # Create template
    cat > "${temp_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${secret_name}
  namespace: ${env}
type: Opaque
stringData:
  # Add your secret keys here
  # Example:
  # database-url: "postgresql://user:pass@host:5432/db"
  # api-key: "your-api-key"
  # jwt-secret: "your-jwt-secret"

  # TODO: Replace with actual secret values
  example-key: "example-value"
EOF

    echo "📝 Created template: ${temp_file}"
    echo "   Opening in editor..."
    echo

    # Open in editor
    ${EDITOR:-vi} "${temp_file}"

    echo
    read -p "Encrypt this secret? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        echo "❌ Cancelled"
        rm "${temp_file}"
        exit 1
    fi

    # Encrypt with SOPS
    echo "🔐 Encrypting secret..."
    sops --encrypt "${temp_file}" > "${output_file}"

    # Remove temp file
    rm "${temp_file}"

    echo "✅ Encrypted secret created: ${output_file}"
    echo
    echo "Next steps:"
    echo "  1. Add to kustomization: edit ${secrets_dir}/kustomization.yaml"
    echo "  2. Uncomment: # - ${secret_name}.yaml"
    echo "  3. Commit: git add ${output_file} && git commit -m 'Add ${secret_name} for ${env}'"
    echo "  4. Push: git push"
}

ENV="$1"
SECRET_NAME="$2"

create_and_encrypt "${ENV}" "${SECRET_NAME}"

System Context

This chapter protects the trust boundary for the rest of the platform.

Secrets discipline matters again in:

• Chapter 04 promotion flows that depend on registry and deploy credentials
• Chapter 10 observability paths that should never leak telemetry secrets
• Chapter 13 guardian analysis, where redaction is mandatory before LLM use

Lab Goal

Run the baseline flow end-to-end:

1. Encrypt secret for develop.
2. Commit/push encrypted manifest.
3. Let Flux decrypt and apply it.
4. Verify secret exists in cluster without exposing values.

Lab file:

• lab.md
• quiz.md

Safe Workflow (Step-by-Step)

1. Verify prerequisites:

command -v sops
command -v age
kubectl get ns flux-system

2. Ensure decryption key exists in cluster:

kubectl -n flux-system get secret sops-age

If missing, create/setup it via:

scripts/sops-setup.sh --create-secret

3. Create encrypted secret manifest:

scripts/sops-encrypt-secret.sh develop backend-secrets

4. Include secret in develop kustomization:

File to edit: flux/secrets/develop/kustomization.yaml

Uncomment:

- backend-secrets.yaml

5. Commit and push:

git add flux/secrets/develop/backend-secrets.yaml flux/secrets/develop/kustomization.yaml
git commit -m "chapter-03: add encrypted backend secret for develop"
git push

6. Verify Flux decrypt/apply:

kubectl -n flux-system get kustomization secrets-develop
kubectl -n flux-system describe kustomization secrets-develop
kubectl -n develop get secret backend-secrets

CI/Logs Guardrails for Secrets

• Never print secret values in CI (echo $SECRET is forbidden).
• Avoid shell trace mode (set -x) around any secret-handling command.
• Verify secret resource existence only, not values:

kubectl -n develop get secret backend-secrets -o name

• Do not use value-dumping commands in shared logs (kubectl get secret ... -o jsonpath='{.data.*}').

Verification Checklist

• backend-secrets.yaml in Git is encrypted (ENC[...] values).
• secrets-develop Kustomization is Ready.
• backend-secrets exists in namespace develop.
• No plaintext values appear in committed diff.
• Local pre-commit no-secrets check passes before commit.
• Local pre-commit flux-kustomize-validate check passes before commit.

Anti-Patterns

• Committing plaintext then “fixing” with later encryption.
• Sharing age.agekey through chat/email or committing it.
• Reusing one leaked credential across all environments.

Done When

• Learner can explain why Git revert is not enough after secret leak.
• Learner can run encrypt -> commit -> Flux decrypt/apply without plaintext exposure.
• Learner can identify where decryption fails (sops-age, .sops.yaml, or Kustomization wiring).