Chapter 04: Secrets Management (SOPS + Age)
Learning Objectives
By the end of this chapter, you will be able to:
- Encrypt secrets with SOPS and Age before committing to Git
- Trace the Flux decryption flow from encrypted YAML to cluster Secret
- Execute secret rotation after a leak incident
- Explain why
git revertis not remediation for an exposed secret
Start with the video for the concept overview, then work through each lesson section.
Plaintext secrets in Git are a production incident waiting to happen. In this chapter, we implement a secure GitOps secret workflow using SOPS and Age to ensure that sensitive data remains encrypted throughout its lifecycle.
1. The Problem: The Durable Leak
A teammate commits a plaintext API key to fix a failing deploy. Reverting the commit is not enough; the secret is already exposed in the Git history and CI logs. This creates a high-pressure situation requiring immediate rotation and auditing across the entire fan-out surface.
2. The Concept: Encrypted-at-Rest GitOps
We use SOPS (Mozilla Secrets Operations) and Age to keep our secrets safe. Secrets are encrypted on the developer’s machine before they are committed, and Flux decrypts them inside the cluster using a private key stored in a sops-age secret.
3. The Code: Secret Structure & Policies
Our sre/ repo defines exactly how and which files should be encrypted. The .sops.yaml policy ensures that only the relevant data fields are encrypted, keeping the YAML metadata visible for auditing.
SOPS encryption policy
creation_rules:
# Cloudflare API token (cert-manager + external-dns)
- path_regex: flux/secrets/cloudflare/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Auth secrets (Dex OIDC)
- path_regex: flux/secrets/auth/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Observability secrets (shared)
- path_regex: flux/secrets/observability/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Development environment secrets
- path_regex: flux/secrets/develop/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Staging environment secrets
- path_regex: flux/secrets/staging/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Production environment secrets
- path_regex: flux/secrets/production/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Local (kind) profile - the learner's own key. Placeholder until
# `scripts/sops-setup.sh --local` writes your public key here.
- path_regex: flux/secrets/local/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1REPLACE_WITH_YOUR_PUBLIC_KEY_RUN_scripts_sops-setup.sh_--local
4. The Guardrail: Secret Encryption Helper
To make the “Safe Path” the easiest path, we use a helper script that automates the encryption process, preventing accidental plaintext commits.
Secret encryption helper
Show the secret encryption helper
#!/bin/bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
usage() {
cat <<EOF
Usage: $0 ENVIRONMENT SECRET_NAME [NAMESPACE]
Create and encrypt a Kubernetes Secret with SOPS
ARGUMENTS:
ENVIRONMENT Target directory under flux/secrets/: develop, staging,
production, or local (the kind profile - encrypted with
your own key, see scripts/sops-setup.sh --local)
SECRET_NAME Name of the secret (e.g., backend-secrets)
NAMESPACE Namespace the Secret is created in (default: ENVIRONMENT,
or develop when ENVIRONMENT is local)
EXAMPLES:
# Create encrypted secret for develop environment
$0 develop backend-secrets
# Create encrypted secret for production
$0 production backend-secrets
# Local kind profile: encrypted with your key, applied to develop
$0 local lab-secret
WORKFLOW:
1. Creates a plaintext secret template
2. Opens it in \$EDITOR (or vi)
3. After you save and quit, encrypts it with SOPS
4. Saves encrypted version to flux/secrets/ENVIRONMENT/
NOTE:
- Make sure .sops.yaml is configured with age public key
- Make sure SOPS and age are installed
- The plaintext file is automatically deleted after encryption
EOF
}
create_and_encrypt() {
local env="$1"
local secret_name="$2"
local namespace="${3:-$env}"
[[ "${env}" == "local" && -z "${3:-}" ]] && namespace="develop"
local secrets_dir="${REPO_ROOT}/flux/secrets/${env}"
local output_file="${secrets_dir}/${secret_name}.yaml"
local temp_file="${secrets_dir}/${secret_name}.yaml.tmp"
# Validate environment
if [[ ! -d "${secrets_dir}" ]]; then
echo "❌ Invalid environment: ${env}"
echo " Available: develop, staging, production, local"
exit 1
fi
# Check if secret already exists
if [[ -f "${output_file}" ]]; then
echo "⚠️ Secret already exists: ${output_file}"
read -p " Edit existing secret with SOPS? (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
sops "${output_file}"
echo "✅ Secret updated"
exit 0
else
exit 1
fi
fi
# Create template
cat > "${temp_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
name: ${secret_name}
namespace: ${namespace}
type: Opaque
stringData:
# Add your secret keys here
# Example:
# database-url: "postgresql://user:pass@host:5432/db"
# api-key: "your-api-key"
# jwt-secret: "your-jwt-secret"
# TODO: Replace with actual secret values
example-key: "example-value"
EOF
echo "📝 Created template: ${temp_file}"
echo " Opening in editor..."
echo
# Open in editor
${EDITOR:-vi} "${temp_file}"
echo
read -p "Encrypt this secret? (y/N): " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "❌ Cancelled"
rm "${temp_file}"
exit 1
fi
# Encrypt with SOPS
echo "🔐 Encrypting secret..."
# The template ends in .tmp, so sops would guess "binary" and emit JSON
# with the whole document in one blob - not a Secret manifest Flux can
# apply. Force YAML in and out.
sops --encrypt --input-type yaml --output-type yaml "${temp_file}" > "${output_file}"
# Remove temp file
rm "${temp_file}"
echo "✅ Encrypted secret created: ${output_file}"
echo
echo "Next steps:"
echo " 1. Add to kustomization: edit ${secrets_dir}/kustomization.yaml"
echo " 2. Uncomment: # - ${secret_name}.yaml"
echo " 3. Commit: git add ${output_file} && git commit -m 'Add ${secret_name} for ${env}'"
echo " 4. Push: git push"
}
ENV="$1"
SECRET_NAME="$2"
NAMESPACE_ARG="${3:-}"
create_and_encrypt "${ENV}" "${SECRET_NAME}" "${NAMESPACE_ARG}"
5. Verification: Did I Get It?
Create an encrypted secret with your key (registered in Chapter 00) and verify that Flux applies it:
scripts/sops-encrypt-secret.sh local lab-secret # encrypted for you, applied to develop
# add it to flux/secrets/local/kustomization.yaml, commit, push, then:
flux reconcile kustomization secrets-local --with-source
kubectl -n develop get secret lab-secret
The platform’s own secrets under flux/secrets/develop/ are encrypted for the platform key; you cannot decrypt them and you do not need to. Your lane is flux/secrets/local/.