Appendix: DNS and TLS Automation

Incident Hook

A service works through a raw load balancer IP, but the real hostname fails during rollout. DNS still points at the wrong target, or Cloudflare answers 526 Invalid SSL certificate, and the incident looks like an application bug. Time is wasted debugging pods while the actual failure sits at the edge. Production ingress needs automated DNS and automated certificates together.

Why This Appendix Exists

The main course keeps early chapters focused on platform safety and GitOps. This appendix explains the edge automation layer used by the SafeOps platform:

• external-dns manages Cloudflare DNS records from cluster state
• cert-manager with Cloudflare’s origin-ca-issuer issues the origin certificates
• Traefik ingresses request a certificate by naming the issuer and their TLS hosts

This is not a separate core chapter because ingress is not the center of the course. It is a supporting production capability you will rely on once the platform is running. It is Cloud track only: on kind there is no public address, so none of this is installed there.

SafeOps Baseline

Traffic takes two TLS hops: visitor → Cloudflare edge → your origin (Traefik).

• DNS: external-dns (namespace external-dns) publishes records only for Ingresses that opt in with the annotation cloudflare-proxied: "enabled", and publishes them as proxied records. It owns its records through TXT records (txtOwnerId k8s-external-dns-<cluster>) and never changes records it does not own - for example those left by an older cluster.
• TLS at the edge: visitors get Cloudflare’s certificate. Nothing to manage in the cluster.
• TLS at the origin: the zone SSL mode is Full (strict) (managed in Terraform), so Cloudflare validates the origin certificate. The origin certificates come from Cloudflare Origin CA through ClusterOriginIssuer cloudflare-origin-ca. Cloudflare trusts them and browsers do not, so they only work behind the proxy. They have no Let’s Encrypt rate limits, however often the cluster is rebuilt.
• Tokens: two separate SOPS secrets with separate scopes. external-dns reads a DNS-only token. origin-ca-issuer (namespace cert-manager) reads a token that can only issue origin certificates (Zone / SSL and Certificates / Edit).
• Monitoring: Prometheus scrapes cert-manager and alerts before a certificate expires (CertificateExpiringSoon, CertificateExpiresIn3Days) or when one stays not ready (CertificateNotReady).

Investigation Snapshots

Here is the DNS/TLS GitOps bundle used in the SafeOps system.

Here is the external-dns release used to synchronize DNS records.

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: external-dns
  namespace: flux-system
spec:
  interval: 30m
  chart:
    spec:
      chart: external-dns
      version: "1.23.0"
      sourceRef:
        kind: HelmRepository
        name: external-dns
        namespace: flux-system
  # Own namespace and a plain release name; before, it ran as cert-manager/cert-manager-external-dns.
  targetNamespace: external-dns
  releaseName: external-dns
  install:
    createNamespace: false
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  values:
    provider:
      name: cloudflare
    env:
      - name: CF_API_TOKEN
        valueFrom:
          secretKeyRef:
            name: cloudflare-api-token
            key: api-token
    extraArgs:
      - --annotation-filter=cloudflare-proxied=enabled
      - --cloudflare-proxied
      - --kube-api-request-timeout=60s
      - --cloudflare-dns-records-per-page=500
    # A and AAAA: a dual-stack load balancer has an IPv6 address too, and its record is published like the
    # IPv4 one (the default list today - named here, so a default change cannot drop IPv6).
    managedRecordTypes: [A, AAAA, CNAME]
    domainFilters:
      - safeops.work
    policy: sync
    sources:
      - ingress
    # external-dns changes only records with its own owner ID (a TXT record next to each). One per
    # provider (cluster-config cloud_provider, written by each Terraform module): cluster_name is "sre"
    # everywhere, and two clusters with one ID rewrote each other's records every minute (Hetzner + GKE,
    # 2026-10-16). A record another owner left is skipped - flux-pre-destroy.sh removes the Ingresses
    # and waits for external-dns to delete their records before a cluster goes.
    txtOwnerId: "k8s-external-dns-${cluster_name}-${cloud_provider}"

Here is the ClusterOriginIssuer that issues origin certificates from Cloudflare Origin CA.

---
# Issues origin certificates for every Ingress through Cloudflare's Origin CA. Works only behind the
# Cloudflare proxy (cluster-config cloudflare_proxied=enabled, zone SSL mode "Full (strict)"): a browser
# reaching the origin directly would not trust these certificates.
# Token: Cloudflare API token "Zone / SSL and Certificates / Edit" for safeops.work, SOPS secret
# cert-manager/cloudflare-origin-ca-token (flux/secrets/cloudflare).
apiVersion: cert-manager.k8s.cloudflare.com/v1
kind: ClusterOriginIssuer
metadata:
  name: cloudflare-origin-ca
spec:
  requestType: OriginECC
  auth:
    tokenRef:
      name: cloudflare-origin-ca-token
      key: api-token

Here is the frontend ingress pattern that requests its certificate from that issuer.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend
  annotations:
    cert-manager.io/issuer: cloudflare-origin-ca
    cert-manager.io/issuer-kind: ClusterOriginIssuer
    cert-manager.io/issuer-group: cert-manager.k8s.cloudflare.com
    cloudflare-proxied: "${cloudflare_proxied}"
spec:
  ingressClassName: traefik
  rules:
  - host: frontend.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 8080

Operating It

Everything here needs the Hetzner cluster: on kind there is no public address and none of these components are installed. Every command names the cluster with --context (Chapter 01).

Safe Workflow (Step-by-Step)

1. Set the zone SSL mode to Full (strict) (Terraform), never Flexible: Flexible sends traffic from Cloudflare to your origin in plain text.
2. Confirm both token Secrets exist, each with the key api-token (both SOPS files live in flux/secrets/cloudflare/):
   • cloudflare-api-token in namespace external-dns - the DNS token (external-dns-cloudflare-api-token.yaml)
   • cloudflare-origin-ca-token in namespace cert-manager - the Origin CA token (cloudflare-origin-ca-token.yaml)
3. Reconcile the dns-and-certificates bundle so external-dns and the issuer exist before you depend on them.
4. Verify that the cert-manager, origin-ca-issuer and external-dns pods are healthy and that the ClusterOriginIssuer is ready.
5. Add or verify the ingress hostnames, the TLS block and the three issuer annotations, exactly as in flux/apps/frontend/base/ingress.yaml:

   cert-manager.io/issuer: cloudflare-origin-ca
   cert-manager.io/issuer-kind: ClusterOriginIssuer
   cert-manager.io/issuer-group: cert-manager.k8s.cloudflare.com
   

6. Wait for the DNS record and the certificate before declaring the route healthy. With the zone on Full (strict), Cloudflare answers 526 while the origin has no certificate it can validate - on a new cluster that is usual until cert-manager has issued one. Watch kubectl --context hetzner-sre-control-plane get certificate -A instead of retrying the URL.
7. Validate with the real hostname over HTTPS, not only with raw service or load balancer IP checks.

Verification Commands

kubectl --context hetzner-sre-control-plane -n cert-manager get pods
kubectl --context hetzner-sre-control-plane -n external-dns get pods
kubectl --context hetzner-sre-control-plane -n external-dns get secret cloudflare-api-token -o jsonpath='{.data}' | jq 'keys'        # ["api-token"]
kubectl --context hetzner-sre-control-plane -n cert-manager get secret cloudflare-origin-ca-token -o jsonpath='{.data}' | jq 'keys'  # ["api-token"]
kubectl --context hetzner-sre-control-plane get clusteroriginissuer
kubectl --context hetzner-sre-control-plane -n develop get certificate,secret
kubectl --context hetzner-sre-control-plane -n develop describe ingress frontend
kubectl --context hetzner-sre-control-plane -n external-dns logs deploy/external-dns --since=10m
curl -fsSI https://develop.safeops.work   # -f: a 5xx (526) or TLS/DNS error fails the command

Common Failure Patterns

• 526 from Cloudflare: with the zone on Full (strict), Cloudflare cannot validate the origin’s certificate - it is missing, not yet issued (usual on a new cluster until cert-manager has issued it), or the Ingress does not reference the issuer.
• Token secret missing or wrong: DNS records or certificates never appear. Check the external-dns logs and the CertificateRequest events.
• Stale DNS record: a record left by an older cluster is not owned by the current external-dns, so it is never updated. Delete it by hand.
• Ingress without cloudflare-proxied: "enabled": external-dns ignores it, and the hostname never resolves.
• Testing only with raw IPs misses that the real hostname path is still broken.

Guardrail Principle

Automate DNS and TLS together, and keep each token to one job. Manual DNS records plus manual certificate handling create hidden outage debt. An expiry alert is part of the automation: a renewal that silently stops is an outage with a date on it.

Done When

• external-dns is reconciling without errors
• the ClusterOriginIssuer is ready and every Ingress certificate is Ready=True
• ingress resources request TLS explicitly from the issuer
• the hostname resolves (proxied) and HTTPS succeeds with the zone on Full (strict)
• you can explain whether a failure belongs to app routing, DNS sync, or certificate issuance