ksail.prod.yaml

# Prod Talos + Hetzner cluster.
# See docs/TEMPLATING.md for the fields a fork of this repo needs to edit.
---
apiVersion: ksail.io/v1alpha1
kind: Cluster
metadata:
  name: prod
spec:
  cluster:
    distributionConfig: talos
    connection:
      context: admin@prod
      timeout: 30m
    distribution: Talos
    provider: Hetzner
    cni: Cilium
    gitOpsEngine: Flux
    # KSail manages the Cluster Autoscaler natively when
    # autoscaler.node.enabled is set. KSail installs, configures, and
    # reconciles the autoscaler — no Flux manifests needed. Initial
    # static node counts still apply on `cluster create`.
    #
    # Cost-optimized overflow: one pool per cost-optimized Hetzner x86
    # shared-vCPU type (the CX line), so the autoscaler can pick the smallest
    # *adequate* node for whatever can't schedule on the static workers. Sizes
    # double cleanly per step (cx43 8c/16G → cx53 16c/32G;
    # ~€8.11/€14.99 gross/mo in fsn1). All pools scale to zero (min: 0) so idle
    # capacity costs nothing; the static workers are the guaranteed baseline and
    # the only Longhorn storage nodes (autoscaler nodes are compute-only).
    # CAX/ARM is cheaper but needs an ARM Talos image — not applicable here.
    #
    # cx33 (4c/8G) is deliberately EXCLUDED as an overflow type. The per-node
    # kube-system + observability DaemonSets (cilium, coroot, kubescape,
    # longhorn-manager, tetragon, …) request ~3Gi combined — already ~40% of an
    # 8G cx33's allocatable — so a cx33 overflow node sits above the 50%
    # scale-down-utilization threshold on DaemonSets ALONE and can NEVER be
    # reclaimed (it stays pinned indefinitely, and on a maxed hcloud account
    # that starves the Talos-snapshot slot). cx43 (16G) amortizes that fixed
    # overhead to ~19%, so an idle overflow node can actually scale back down.
    #
    # expander: [LeastNodes, LeastWaste] — an ordered priority chain (upstream
    # cluster-autoscaler --expander=least-nodes,least-waste). LeastNodes runs
    # FIRST: on scale-up it keeps the node groups that meet the pending pods
    # with the FEWEST total new nodes (preferring the largest adequate type,
    # cx53 > cx43, to consolidate a burst onto fewer, bigger servers).
    # LeastWaste is the tiebreaker: among the groups LeastNodes leaves tied,
    # pick the one with the least idle CPU/memory.
    # NB this list form requires KSail's expander-list support ("feat: support a
    # priority list of expanders"); the scalar-only releases up to v7.57.0
    # reject a list, so the pinned KSAIL_VERSION (ci.yaml/cd.yaml) must be bumped
    # to a release that ships it before this validates / deploys.
    # "Price" would be the direct cost optimizer, but the Hetzner
    # cluster-autoscaler provider does not implement the pricing API, so KSail
    # rejects Price+Hetzner (and the autoscaler would crash on startup).
    #
    # maxNodesTotal is the HARD CEILING ON TOTAL CLUSTER NODES, INCLUDING the
    # static baseline. 9 = the Hetzner account cap (10) MINUS ONE RESERVED SLOT:
    # 6 static (3 control-planes + 3 workers) + up to 3 autoscaler nodes. KSail
    # passes it to cluster-autoscaler --max-nodes-total, which counts every node,
    # so this total framing already matches the runtime. Each pool max is 4, so
    # any single CX type can still fill the autoscaler headroom (or any mix); the
    # cluster total stays <= 9.
    #
    # THE RESERVED SLOT IS LOAD-BEARING — DO NOT RAISE THIS TO THE ACCOUNT CAP.
    # `deploy-prod` ensures the autoscaler config secret on EVERY deploy, and that
    # step builds the Talos snapshot whenever one is absent (e.g. after any Talos
    # version bump). The build needs a TEMPORARY hcloud server, so it needs a free
    # slot. At maxNodesTotal == serverLimit a fully scaled-out cluster leaves none,
    # the build fails `resource_limit_exceeded`, and because the ensure step runs
    # regardless of what a PR changed, EVERY merge_group deploy is evicted AND
    # `Heal Prod` fails too — wedging the whole queue with no CD path to prod
    # (platform#2419: v1.13.5 on 07-03, recurring identically for v1.13.6 on 07-16).
    #
    # KSail now ENFORCES this: ksail#6172 rejects a Talos+Hetzner config whose
    # reachable total equals serverLimit (`ErrAutoscalerLeavesNoSnapshotSlot`), and
    # pins one-below-the-limit as the valid boundary. Only Talos reserves a slot —
    # other distributions never build a snapshot, so they keep the full limit.
    # If peak capacity ever needs the 10th node, raise the hcloud account limit
    # FIRST and keep maxNodesTotal strictly below it.
    #
    # NB KSail's config validation is being changed (ksail#5017) to expect
    # maxNodesTotal as this total (validate maxNodesTotal <= serverLimit). Until
    # that ships, the old validation computes controlPlanes + workers +
    # min(maxNodesTotal, Σ pool.max) = 3 + 3 + 10 = 16 and would reject this, so
    # the KSail change must land before this deploys. (Scale-up is in any case
    # currently blocked by the user_data limit, ksail#5015.)
    autoscaler:
      node:
        enabled: true
        expander: [LeastNodes, LeastWaste]
        maxNodesTotal: 9
        scaleDownUnneededTime: "10m"
        # Native over-provisioning: install the CapacityBuffer CRD + controller
        # so the platform's CapacityBuffer (k8s/providers/hetzner/infrastructure/
        # overprovisioning/) reserves warm scale-up headroom as virtual,
        # pod-less chunks — the native replacement for the old balloon-pod
        # ReplicaSet + cluster-proportional-autoscaler. Ignored unless the node
        # autoscaler is installed (it is, above). Requires a KSail release that
        # ships ksail#5277 (post-v7.58.3); the CI/CD KSAIL_VERSION pin
        # (renovate-managed) must reach it first.
        capacityBuffers: true
        pools:
          - name: autoscale-cx43
            serverType: cx43
            location: fsn1
            min: 0
            max: 4
          - name: autoscale-cx53
            serverType: cx53
            location: fsn1
            min: 0
            max: 4
    # These components are installed by ksail. There is a known race on
    # Hetzner clusters between ksail's metrics-server install and Flux's
    # kubelet-serving-cert-approver install (required because Talos
    # rotate-server-certificates: true leaves kubelet serving CSRs
    # pending). File upstream in devantler-tech/ksail if this trips
    # bootstrap again.
    certManager: Enabled
    metricsServer: Enabled
    policyEngine: Kyverno
    controlPlanes: 3
    workers: 3
    # Pin Kubernetes deliberately so 'ksail cluster update' doesn't roll nodes
    # to ksail's newer default when that default outruns what the pinned Talos
    # version supports (rejected with "kubelet image is not valid: version of
    # Kubernetes ... is too new"). Talos v1.13.5 supports Kubernetes 1.31-1.36;
    # bump in lockstep with Talos rather than letting the default drift.
    #
    # In-place Pod resize (InPlacePodVerticalScaling, beta and ON by default
    # from k8s 1.33) lets VPA -- already 1.6 with updateMode InPlaceOrRecreate
    # -- resize pods, including the kube-system datapath now under auto-vpa, in
    # place instead of evicting them. Upgraded one minor at a time
    # (1.32 -> 1.33 -> 1.34 -> 1.35 -> 1.36); now on 1.36.2. Talos v1.13.5's
    # ceiling is Kubernetes 1.36.
    kubernetesVersion: v1.36.2
    talos:
      # Pin Talos to match the ISO so 'ksail cluster update' doesn't attempt
      # an unwanted in-place upgrade to ksail's default version.
      version: v1.13.6
      # Hetzner Talos ISO (x86_64). The v1.11.2 ISO (122630) was deprecated
      # and removed from Hetzner on 2026-03-18. Use v1.12.4 (125127).
      iso: 125127
      # Talos Image Factory extensions baked into the node image. KSail
      # computes the schematic ID from this list and uses it both for
      # machine.install.image (derived during config generation — there is no
      # hand-maintained install-image patch) and for the Hetzner snapshot the
      # Cluster Autoscaler boots new nodes from.
      #   - siderolabs/iscsi-tools      (Longhorn data plane)
      #   - siderolabs/util-linux-tools (Longhorn fstrim)
      #   - siderolabs/qemu-guest-agent (Hetzner Cloud VM integration)
      extensions:
        - siderolabs/iscsi-tools
        - siderolabs/util-linux-tools
        - siderolabs/qemu-guest-agent
    localRegistry:
      registry: "devantler:${GHCR_TOKEN}@ghcr.io/devantler-tech/platform/manifests"
    # Cosign keyless signature verification on the flux-system OCIRepository that
    # KSail generates and owns (rendered onto spec.verify; a hand-written override
    # would lose the reconcile fight, so it is configured here per ksail#4987).
    # The manifests artifact is already cosign-signed keyless in the deploy-prod
    # composite (push → sign → attest); this makes Flux REJECT any infra artifact
    # whose signature is missing or whose signer is not the platform deploy
    # workflow — closing the supply-chain gap #1559 opened for apps but left on
    # the infra source (#1570). The signer identity + ref were confirmed against
    # the live signature with `cosign verify`: the artifact is signed by
    # .github/workflows/ci.yaml on refs/heads/gh-readonly-queue/main/<pr> (the
    # merge-queue deploy-prod path — the real prod deploy) under GitHub's Fulcio
    # OIDC issuer. The subject is pinned to the two *production* deploy refs only
    # (merge-queue refs off main, plus a manual cd.yaml dispatch from main) rather
    # than `@.+`, so a signature these workflows might produce on any other ref is
    # not trusted (defence in depth on a high-blast-radius source).
    verify:
      provider: cosign
      matchOIDCIdentity:
        # Automatic prod deploy: ci.yaml deploy-prod runs on merge_group, which
        # signs under the merge-queue ref refs/heads/gh-readonly-queue/main/<pr>.
        - issuer: '^https://token\.actions\.githubusercontent\.com$'
          subject: '^https://github\.com/devantler-tech/platform/\.github/workflows/ci\.yaml@refs/heads/gh-readonly-queue/main/.+$'
        # Manual prod deploy: cd.yaml is workflow_dispatch, run from main.
        - issuer: '^https://token\.actions\.githubusercontent\.com$'
          subject: '^https://github\.com/devantler-tech/platform/\.github/workflows/cd\.yaml@refs/heads/main$'
        # Disaster recovery: dr-rebuild.yaml is workflow_dispatch, run from main,
        # and republishes this same artifact while rebuilding prod from zero.
        # Without this entry a recovery produces a correctly-signed artifact that
        # Flux still refuses — the one situation where a manual workaround is
        # least available. Pinned to refs/heads/main, like cd.yaml above, so a DR
        # run dispatched from any other ref is not trusted.
        - issuer: '^https://token\.actions\.githubusercontent\.com$'
          subject: '^https://github\.com/devantler-tech/platform/\.github/workflows/dr-rebuild\.yaml@refs/heads/main$'
  provider:
    hetzner:
      # Hetzner account hard cap: at most 10 servers total. This mirrors the
      # account's real limit, so it tracks the account — not the cluster budget.
      # maxNodesTotal is deliberately held STRICTLY BELOW it (9 < 10) to reserve a
      # slot for the deploy-time Talos snapshot build's temporary server; ksail#6172
      # enforces that for Talos+Hetzner. See the autoscaler block for why the
      # reservation is load-bearing (platform#2419). If this cap is ever raised,
      # keep maxNodesTotal strictly below it.
      serverLimit: 10
      controlPlaneServerType: cx33
      # Baseline workers on cx43 (8c/16G) rather than cx33 (4c/8G). These 3
      # static workers carry the platform's full memory footprint plus the
      # ~2 GB/node fixed per-node agent tax (Cilium, Longhorn, coroot/kubescape
      # node-agents). On 8 GB nodes that tax + VPA's peak-sized (correct,
      # OOM-safe) memory requests saturated node *requests* to ~99% while real
      # usage sat ~50%, leaving no reschedule headroom and forcing ~2-3m
      # autoscaler cold-boots on every reschedule (the page-timeout flap). 16 GB
      # workers amortize the fixed tax over more capacity and let VPA apply its
      # requests without eviction churn. Control planes stay cx33 (their load
      # is steady and well within 8 GB).
      workerServerType: cx43
      location: fsn1
      # Stable API endpoint (#2120): provision a Hetzner floating IP and render
      # it as the Kubernetes/Talos API endpoint + certificate SANs, plus a
      # control-plane Talos VIP block so the elected leader owns the address.
      # The existing control-plane node IPs stay in the SANs, so current
      # KUBE_CONFIG/TALOS_CONFIG secrets keep working; a CP-node recreate no
      # longer breaks the endpoint (DR runbook Scenario 9). floatingIPLocation
      # defaults to the cluster location (fsn1). Applying rolls the CP machine
      # configs (VIP + SANs); ksail >= v7.120.0 required (CI pins 7.156.0).
      floatingIPEnabled: true
      networkCidr: 10.0.0.0/16
      placementGroupStrategy: Spread
      fallbackLocations:
        - nbg1
        - hel1
      placementGroupFallbackToNone: true
  workload:
    sourceDirectory: k8s
    kustomizationFile: clusters/prod
    tag: latest
    validation:
      # Parity with ksail.yaml: the prod kustomization pulls the same shared
      # bases (Coroot + Flagger), so `ksail --config ksail.prod.yaml workload
      # validate` hits the identical CRDs whose CRD-catalog schemas kubeconform
      # cannot resolve (requires ksail >= 7.26.0):
      #   - Coroot (coroot.com/v1): the datreeio catalog schema is stale
      #     (additionalProperties: false, missing newer fields like
      #     nodeAgent/clickhouse), so kubeconform rejects valid manifests.
      #   - MetricTemplate / Canary (flagger.app/v1beta1): the Flagger CRDs are
      #     installed by the flagger HelmRelease at runtime and are not in the
      #     catalog, so the MetricTemplates and the onboarded Canaries (umami,
      #     homepage, opencost) can't be schema-checked offline.
      #   - Tenant (kro.run/v1alpha1): the CRD is generated at runtime by the
      #     KRO controller from the tenant ResourceGraphDefinition (#1932), so
      #     no static schema exists in any catalog (same class as the Flagger
      #     runtime CRDs). Prod renders no Tenant CR yet (the pilot is local
      #     opt-in), but `ksail workload validate` schema-checks unreferenced
      #     files standalone, so both configs need the skip.
      # Validation-only: the prod CD path is `workload push` (no validateOnPush),
      # so this field never affects what is deployed. Remove once the upstream
      # Coroot/Flagger CRD catalog schemas are current (drop from both configs).
      skipKinds:
        - Coroot
        - MetricTemplate
        - Canary
        - Tenant