fix(external-dns): harden single-replica writer against node failure (#2128)

Coroot flags external-dns:Deployment as "single instance - not resilient
to node failure". external-dns v0.21.0 (chart 1.21.1) has NO leader
election — the multi-replica proposal is status `not-planned` and the
`--enable-leader-election` flag does not exist in the binary — so a 2nd
ACTIVE replica is unsafe: both write to Cloudflare under one txtOwnerId,
causing duplicate writes / rate-limit churn / records fighting. The
validate-replica-floor policy exempts external-dns for this reason.

Instead of a hazardous duplicate-writes 2-replica config, minimise the
node-failure blast radius of the deliberately-single writer:
- priorityClassName: system-cluster-critical so the replacement pod
  reschedules ahead of normal workloads on node loss (matches its peers
  CoreDNS and the hcloud-csi controller), shrinking the DNS-drift window.
- drain-safe PodDisruptionBudget (maxUnavailable: 1, the #1880 pattern;
  chart ships no PDB template) so a node drain cycles the single pod
  cleanly instead of stalling.
- document the TXT-registry ownership fence that makes a reschedule safe
  and the upgrade path to true active/standby HA once external-dns ships
  leader election.

Validated with `kubectl kustomize k8s/providers/hetzner/infrastructure/external-dns`.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: devantler

k8s/providers/hetzner/infrastructure/external-dns/helm-release.yaml

@@ -22,6 +22,13 @@ spec:
     sources:
       - gateway-httproute
     policy: sync
+    # TXT-registry ownership is what makes this single writer safe to reschedule:
+    # every managed record gets a paired "_externaldns." TXT carrying this
+    # txtOwnerId, so a pod that restarts on another node (node failure / drain)
+    # recognises the records it already owns and reconciles idempotently instead
+    # of re-creating or fighting them. This ownership fence is also why a second
+    # ACTIVE replica is unsafe (see priorityClassName note) — two instances would
+    # share one owner id and issue duplicate Cloudflare writes.
     registry: txt
     txtOwnerId: "${domain}"
     txtPrefix: "_externaldns."
@@ -36,6 +43,36 @@ spec:
           secretKeyRef:
             name: external-dns-cloudflare
             key: api-token
+    # Node-failure resilience for a DELIBERATELY single-replica writer.
+    #
+    # external-dns has NO leader election at the pinned app version (chart 1.21.1
+    # = external-dns v0.21.0). The "support multiple replicas with leader
+    # election" proposal (docs/proposal/001-leader-election.md) is status
+    # `not-planned` and the `--enable-leader-election` flag does not exist in the
+    # controller binary — the only concurrency control is `--once` (run-once).
+    # So running >1 ACTIVE replica is unsafe: every replica would write to
+    # Cloudflare under the same txtOwnerId, causing duplicate API calls,
+    # rate-limit churn, and records fighting each other. The validate-replica-floor
+    # policy exempts external-dns by name for exactly this reason.
+    #
+    # Therefore we do NOT run 2 always-active replicas. We instead minimise the
+    # node-failure blast radius of the single writer:
+    #   * priorityClassName: system-cluster-critical — external-dns programs the
+    #     cluster's public DNS (its peer infrastructure CoreDNS and the hcloud-csi
+    #     controller use the same class), so on a node loss the scheduler places
+    #     the replacement pod ahead of normal workloads, shrinking the window in
+    #     which DNS records drift from desired state.
+    #   * a drain-safe PDB (pod-disruption-budget.yaml, maxUnavailable: 1) so a
+    #     node drain cycles the single pod cleanly instead of being a surprise.
+    #   * TXT-registry ownership (above) so the rescheduled pod reclaims its
+    #     records idempotently.
+    #
+    # UPGRADE PATH to true active/standby HA: once external-dns ships leader
+    # election (the `--enable-leader-election` flag, currently `not-planned`),
+    # switch to replicas: 2 with leader election enabled (only the leader writes),
+    # a topologySpreadConstraints across nodes, and drop this single-replica note.
+    # Track the upstream proposal before bumping the replica count.
+    priorityClassName: system-cluster-critical
     deploymentStrategy:
       type: Recreate
     serviceMonitor:

k8s/providers/hetzner/infrastructure/external-dns/kustomization.yaml

@@ -7,3 +7,4 @@ resources:
   - helm-release.yaml
   - helm-repository.yaml
   - networkpolicy.yaml
+  - pod-disruption-budget.yaml

k8s/providers/hetzner/infrastructure/external-dns/pod-disruption-budget.yaml

@@ -0,0 +1,20 @@
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: external-dns
+  namespace: external-dns
+spec:
+  # maxUnavailable: 1 is the platform-wide drain-safe PDB pattern (issue #1880):
+  # it never deadlocks a node drain regardless of replica count. external-dns is
+  # a DELIBERATELY single-replica writer (no leader election at the pinned app
+  # version v0.21.0 — see helm-release.yaml; a 2nd ACTIVE replica would issue
+  # duplicate Cloudflare writes under one txtOwnerId), so this PDB does not add
+  # standby redundancy. Its job is to let a node drain evict and reschedule the
+  # single pod cleanly (the high priorityClassName makes that reschedule fast)
+  # rather than the drain stalling on an un-budgeted pod. Convert to a true HA
+  # budget once external-dns ships leader election and this runs replicas: 2.
+  maxUnavailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: external-dns
+      app.kubernetes.io/instance: external-dns