Skip to content

AGENTS.md

Latest commit

 

History

History
372 lines (288 loc) · 20.7 KB

AGENTS.md

File metadata and controls

372 lines (288 loc) · 20.7 KB
Raw

AGENTS.md

This file provides guidance to All Agents when working with code in this repository.

Project Overview

This is a GitOps-managed home Kubernetes cluster using FluxCD and Talos Linux. Runtime application secrets normally come from 1Password via External Secrets (op-secret-store). SOPS is still used for Flux cluster variables, Talos/bootstrap secrets, and existing SOPS-managed resources.

Common Commands

# List all available tasks
task

# Apply a Flux Kustomization manually
task kubernetes:apply-ks cluster=main path=<category>/<app>/app

# Browse a PVC interactively
task kubernetes:browse-pvc ns=<namespace> claim=<pvc-name>

# Delete failed/evicted pods
task kubernetes:delete-failed-pods

# Sync ExternalSecrets
task kubernetes:sync-secrets ns=<namespace> secret=<name>

# Validate a GitOps app without applying it
task kubernetes:validate-app app=<category>/<app>

# Run focused schema validation on YAML files or directories
task kubernetes:validate-yaml path=<path>

# Resolve a mutable image tag to an immutable digest reference
task kubernetes:pin-image image=docker.io/library/nginx:latest

# Check effective workload images for missing digest pins
task kubernetes:check-image-pins

# Resolve the image and record explicit health, storage, database, and auth decisions first
task kubernetes:preflight-new-app app=tools/example args="--image docker.io/library/nginx:latest --port 8080 --no-health-endpoint --persistence none --database none --auth internal"

# Scaffold from the pinned image and probe decision returned by the preflight
task kubernetes:new-app category=tools app=example args="--image docker.io/library/nginx:latest@sha256:<resolved-digest> --port 8080 --no-health-endpoint --internal"

# Upsert one Homarr app and tile. Pipe credentials as JSON; never write them to disk.
credential_provider | task kubernetes:homarr-upsert-app args="--name Example --href https://example.${HOME_DOMAIN} --icon-url https://example/icon.svg"

# Follow updated PR heads, required checks, and merge through an explicit repository target
task github:deliver-pr pr=<number> args="--merge"

# Watch Flux converge a pushed app
task kubernetes:reconcile-app app=tools/searxng

# Start a verified, bounded Alertmanager silence. Routine app deployments and
# internal cluster work stay off the public status page.
export MAINTENANCE_KUBECONFIG="$HOME/.kube/config"
scripts/cluster-maintenance begin \
  --reason "planned cluster maintenance" --duration 4h

# Add --public-status only when user-visible service impact is expected.
# That opt-in requires STATUS_API_KEY from the TheZoo Status item in 1Password.
export STATUS_API_URL="https://status.thezoo.house"
scripts/cluster-maintenance begin \
  --reason "planned public service interruption" --duration 4h --public-status

# Admit the exact known persistent SMART interface-speed baseline while still
# failing closed on every other warning or critical alert.
scripts/cluster-maintenance begin \
  --reason "planned cluster maintenance" --duration 4h \
  --allow-active-alert \
  alertname=SmartDeviceInterfaceSlow,kubernetes_node=k8s-rhea,device=sda

# Renew, inspect, or end the exact silence returned by begin
scripts/cluster-maintenance renew --id <silence-id> --duration 4h
scripts/cluster-maintenance status --id <silence-id>
scripts/cluster-maintenance end --id <silence-id>

Architecture

Directory Structure

  • kubernetes/apps/<category>/<app>/ - Application deployments organized by category (arrs, media, observability, etc.)
  • kubernetes/flux/ - Flux configuration, repositories, and cluster variables
  • kubernetes/templates/ - Reusable templates referenced by app kustomizations
  • kubernetes/bootstrap/ - Initial cluster bootstrap resources
  • talos/ - Talos Linux cluster configuration (talconfig.yaml, patches, secrets)
  • .taskfiles/ - Task automation scripts

Flux GitOps Flow

  1. kubernetes/flux/apps.yaml is the root Kustomization that applies everything under kubernetes/apps/
  2. It enables SOPS decryption via sops-age secret
  3. Variable substitution from: cluster-settings (ConfigMap), cluster-secrets (Secret), cluster-ipam (ConfigMap), volsync-schedules (ConfigMap)
  4. Child Kustomizations inherit decryption and substitution unless labeled substitution.flux.home.arpa/disabled: "true"

App Layout Pattern

Each app follows: kubernetes/apps/<category>/<app>/app/

  • kustomization.yaml - References resources and templates
  • helmrelease.yaml - HelmRelease with chart configuration
  • externalsecret.yaml - Runtime app secrets from 1Password via External Secrets when needed
  • *.sops.yaml - Only for Flux/Talos/bootstrap secrets or existing areas that intentionally use SOPS directly

App Category Placement

  • tools - agent/admin/operator utilities and small cluster helper UIs
  • services - user-facing internal apps that do not fit a narrower domain
  • network - ingress, DNS, routing, SMTP, tunnels, network controllers
  • database - database operators, shared datastores, caches, and database cluster resources
  • observability - monitoring, alerting, logging, dashboards, status checks
  • media - media library, playback, download, or metadata apps
  • arrs - Arr-stack apps and download automation
  • games - game servers and game-specific routers
  • security - identity, auth, policy, and security services

Tooling and Python

  • Use the tools declared in .mise.toml for this repo: kubectl, sops, age, task, flux2, talhelper, talosctl, krew, Renovate, yayamlls, and Python.
  • Python is pinned through Mise for standard-library-only validation helpers. The repository does not declare third-party Python dependencies such as PyYAML.
  • For YAML inspection and transformation, prefer yq, kubectl kustomize, helm template, and the scripts under scripts/.
  • Use yayamlls for focused schema and language-server validation. It does not format YAML; preserve the repository's existing YAML style when editing.
  • If a helper script uses Python, keep it Python standard-library only unless a Python environment and dependency file are added deliberately.
  • Do not install Python packages globally/Homebrew just to inspect YAML. If a one-off Python dependency is unavoidable, use an isolated temporary environment and do not make it part of the repo workflow without adding explicit project config.

YAML Conventions

Schema Headers (required first line)

# HelmRelease
# yaml-language-server: $schema=https://k8s-schemas.pages.dev/helm.toolkit.fluxcd.io/helmrelease_v2.json

# Flux Kustomization
# yaml-language-server: $schema=https://k8s-schemas.pages.dev/kustomize.toolkit.fluxcd.io/kustomization_v1.json

# Plain kustomization.yaml
# yaml-language-server: $schema=https://json.schemastore.org/kustomization

HelmRelease Defaults

  • spec.interval: 10m
  • spec.install.remediation.retries: 3
  • spec.upgrade.cleanupOnFail: true
  • spec.upgrade.remediation.strategy: rollback, retries: 3
  • Skip CRD management unless chart requires it

Style

  • 2-space indentation, no tabs
  • Top-level key order: apiVersion, kind, metadata, spec
  • Use YAML anchors for repeated values (&probes, &cephBucket)
  • Variable substitution uses ${VAR_NAME} syntax (processed by Flux postBuild)
  • End files with single newline

Secrets Management

  • Runtime app secrets should use 1Password plus External Secrets by default:
    • Create or update an item in the Kubernetes vault.
    • Add an ExternalSecret using secretStoreRef.kind: ClusterSecretStore and secretStoreRef.name: op-secret-store.
    • Target the app secret as <app>-secret unless the chart expects a different name.
    • Consume the generated Secret with envFrom.secretRef, explicit env refs, or valuesFrom as appropriate.
  • Use SOPS (*.sops.yaml) for Flux cluster variables, Talos/bootstrap secrets, and existing SOPS-managed resources, not as the default for new app runtime secrets.
  • Never commit plaintext secrets.

Shared Data Services

Prefer existing shared data platforms before adding embedded sidecars or app-local singleton databases.

PostgreSQL

  • Use the existing CloudNativePG/Postgres 17 service for Postgres-compatible apps.
  • Default app connection host: postgres17-pooler.database.svc.cluster.local.
  • Direct writer host is available as postgres17-rw.database.svc.cluster.local when an app requires direct session behavior that the pooler breaks.
  • Initialize app databases/users through GitOps-owned init resources or established app patterns, not manual psql mutations.
  • Store app DB credentials in 1Password and surface them with ExternalSecret via op-secret-store.
  • Common env pattern:
    • DATABASE_URL=postgresql://...@postgres17-pooler.database.svc.cluster.local:5432/<db>
    • INIT_POSTGRES_HOST=postgres17-pooler.database.svc.cluster.local
    • INIT_POSTGRES_DBNAME, INIT_POSTGRES_USER, INIT_POSTGRES_PASS, INIT_POSTGRES_SUPER_PASS

MariaDB/MySQL

  • Use the MariaDB Operator in the database namespace for MariaDB/MySQL-compatible apps.
  • Prefer operator-managed app-specific MariaDB resources through GitOps over embedded database sidecars.
  • Use ceph-block storage unless the app has a documented reason to use a different storage backend.
  • Only use same-pod database sidecars for small apps when no suitable shared/operator path exists, and document why.

Redis-compatible cache/queue

  • Use shared Dragonfly for Redis-compatible needs unless isolation requires otherwise.
  • In-cluster URL pattern: redis://dragonfly.database.svc.cluster.local:6379/<db>.
  • If a workload in another namespace uses Dragonfly and network policy is enforced, update the Dragonfly client NetworkPolicy in GitOps before rollout.

Object storage

  • Prefer existing Rook-Ceph object storage/RGW patterns for S3-compatible needs.
  • Keep browser-facing presigned URLs on a hostname the browser can reach, not an in-cluster service DNS name.
  • Store S3 credentials in 1Password and expose through ExternalSecret.

Talos Configuration

  • Main config: talos/talconfig.yaml
  • Per-node configs: talos/clusterconfig/
  • Patches: talos/patches/global/ and talos/patches/controlplane/
  • Encrypted secrets: talos/talsecret.sops.yaml

Cluster Maintenance Windows

Any planned operation that can roll, drain, reboot, reconcile, or otherwise disrupt live cluster resources must create a bounded maintenance silence immediately before the first live mutation. Use scripts/cluster-maintenance begin; do not construct ad hoc Alertmanager payloads. The helper coordinates concurrent starts with a continuously renewed Kubernetes Lease, then releases it through resource-versioned expiration so an earlier operator cannot delete a successor's lock. Release failures warn without hiding an already verified silence ID, and the Lease ages out safely. The helper fails closed when Alertmanager HA is not ready, an unrelated unsilenced warning or critical alert already exists, another owned maintenance window is active, or both concrete Alertmanager pods do not report the same active silence and expected alert coverage. A known persistent baseline may be admitted with a repeatable exact --allow-active-alert selector containing alertname plus at least one scoping label. The established Renovate baseline is alertname=SmartDeviceInterfaceSlow,kubernetes_node=k8s-rhea,device=sda. This is an admission exception only: keep the alert visible, verify its identity has not changed, and continue to reject every additional warning or critical alert.

The standard matcher covers normal named alerts but deliberately excludes Watchdog and InfoInhibitor, preserving the dead-man heartbeat and null-routed helper path. The default window is four hours and the hard maximum is 24 hours. Routine begin creates only the Alertmanager silence. Do not publish internal implementation work, new app deployments, rehearsals, or changes with no expected user-visible impact. Add --public-status only when users are expected to see an outage or degraded service; that opt-in requires STATUS_API_KEY and creates an identified maintenance incident before the silence. The helper atomically persists its state under ~/.local/state/home-k8s/cluster-maintenance.json (override with MAINTENANCE_STATE_FILE). For a public window, if either side fails, it compensates the side that already succeeded. Renew only the returned silence ID. Call end only after the merged revision, Flux/Tuppr convergence, affected workloads, cluster health, and component-relevant routes or APIs are verified. end refuses to restore alerting or resolve any public notice while unexpected warning or critical alerts remain. It expires the silence before resolving an attached notice. A failed rollout keeps the window active only while a protected GitOps rollback is being executed and verified. If the operator dies, the finite Alertmanager expiry restores alerting automatically while the persisted state preserves any public notice for explicit resolution.

Run scripts/cluster-maintenance probe to exercise Alertmanager's create/read/delete lifecycle with a unique, preflighted alert name. The probe rejects any collision, verifies both members, and expires itself in cleanup even when verification fails.

App-Template (bjw-s) Pattern

When using app-template charts:

  • Use controllers with per-controller containers, probes (anchor &probes), and restrictive securityContext
  • defaultPodOptions.securityContext: runAsNonRoot, runAsUser, runAsGroup, fsGroup, seccompProfile: RuntimeDefault
  • External ingress: ingressClassName: external with external-dns
  • Internal ingress: ingressClassName: internal
  • LoadBalancers: Use Cilium annotation io.cilium/lb-ipam-ips: "${IPAM_IP_*}"

OCI Helm Chart Sources

Use an app-local OCIRepository plus same-namespace HelmRelease.spec.chartRef whenever upstream publishes the exact chart through a reliable public OCI registry. The source and release use the app name, and the OCI source must select the Helm chart content layer:

spec:
  layerSelector:
    mediaType: application/vnd.cncf.helm.chart.content.v1.tar+gzip
    operation: copy
  ref:
    tag: 1.2.3
  url: oci://registry.example/charts/example

Treat HTTP HelmRepository-to-OCIRepository conversion as a source-only handoff, never as a chart upgrade:

  • Add and reconcile the OCIRepository before changing the HelmRelease.
  • Preserve the exact chart version, release identity, values, strategies, hooks, post-renderers, target namespace, and every non-source field.
  • Keep the shared HelmRepository through the observation and rollback window.
  • Run scripts/check-oci-source-handoffs <base-git-revision> before commit. The guard requires matching source versions, no non-source HelmRelease changes, the app-local OCI house pattern, and byte-identical HTTP and OCI chart packages by default. For an explicitly approved rollout-bearing migration, annotate the OCIRepository with oci.home.arpa/allow-rollout: "true"; this is required for OCI-backed HelmRepository conversions and exact-version package mismatches.
  • Unless a rollout is explicitly approved, require no rendered workload change, pod restart, resource replacement, delete/prune operation, or application downtime. An approved rollout still preserves chart versions, release identity, Services, PVCs, data, and every non-source HelmRelease field.
  • Keep HTTP HelmRepository sources when upstream has no reliable exact OCI publication, and document that exception rather than creating a mirror by default.
  • Do not treat an OCI-backed HelmRepository to OCIRepository conversion as source-only. Flux decorates OCIRepository chart versions with OCI digest build metadata, which can change templates that reference .Chart.Version and roll workloads even when the chart archive is identical. Leave these releases on the OCI HelmRepository abstraction unless a separately approved rollout is intended and validated.

Firecrawl chart promotion is intentionally separate from Renovate. The private, signed downstream publisher dispatches .github/workflows/firecrawl-chart-promotion.yaml with an immutable version and digest. That workflow verifies the package and exact OIDC identity, updates only the OCI tag plus its digest comment, opens a ShamuBot pull request, and enables automatic merge behind the protected Static Analysis and Flate checks. A GitHub push webhook then asks Hermes to run scripts/verify-firecrawl-live <main-commit> and close the matching firecrawl-promotion issue only after Flux, Helm, all workloads, the public route, and shared Dragonfly, PostgreSQL, and RabbitMQ connections pass.

New App Validation

Before committing a new or changed app, run:

scripts/validate-app <category>/<app>
# or
task kubernetes:validate-app app=<category>/<app>

This renders the app kustomization, derives its effective namespace from the owning Flux Kustomization, server-dry-runs non-Secret manifests, resolves declared HelmRepository, OCI-backed HelmRepository, GitRepository, and same-namespace OCIRepository chartRef sources, merges ConfigMap and Secret valuesFrom inputs in Flux order, renders HelmReleases with common Flux substitutions, rejects malformed chart documents, checks effective PodSpec images for immutable digests, and server-dry-runs the rendered chart output. Raw Secret manifests are skipped during local dry-run because SOPS material may only be available to Flux. Use scripts/validate-app --offline <category>/<app> for credential-free rendering; required valuesFrom resources must then be present in the rendered app context.

Repository-wide image validation uses both scripts/check-image-pins kubernetes/apps for rendered Kustomize workloads and scripts/check-helmrelease-images for Helm-rendered workloads. The Helm check freezes existing legacy tag-only chart defaults in scripts/helm-image-pin-baseline.txt; new entries should be digest-pinned rather than added to the baseline. Offline render gaps must be explicit and reasoned in scripts/helm-image-validation-skips.txt.

For public images, pin mutable tags before committing:

scripts/pin-image docker.io/searxng/searxng:latest
# or
task kubernetes:pin-image image=docker.io/searxng/searxng:latest

Before scaffolding a new standard app-template app, make the runtime contract explicit and resolve the mutable image to an immutable digest:

scripts/preflight-new-app tools/example \
  --image docker.io/library/nginx:latest \
  --port 8080 \
  --no-health-endpoint \
  --persistence none \
  --database none \
  --auth internal

When upstream source is available, pass --source <checkout> --strict-evidence to require source evidence for the declared port, health endpoint, persistence paths, and database choice. The report includes image platforms, entrypoint, command, exposed ports, declared volumes, and environment variable names when crane is available. Unknowns are warnings rather than guessed values.

Then scaffold the app with the pinned image returned by the preflight:

scripts/new-app tools example --image docker.io/library/nginx:latest@sha256:<resolved-digest> --port 8080 --health-path / --internal
# or
task kubernetes:new-app category=tools app=example args="--image docker.io/library/nginx:latest@sha256:<resolved-digest> --port 8080 --health-path / --internal"

For focused YAML validation, scripts/validate-yaml automatically recognizes a single kubernetes/apps/<category>/<app> root and runs the canonical scripts/validate-app --offline workflow. This avoids reading whichever Kubernetes context happens to be active. Set VALIDATE_YAML_APP_MODE=online only when cluster-backed substitution and server-side dry-run are intentional. Exit code 201 from yayamlls is explained as a schema failure or unsupported schema instead of being returned without context.

For Homarr v1 dashboard updates, use scripts/homarr-upsert-app. It reads { "username": "...", "password": "..." } from stdin, owns and closes its local port-forward unless --base-url is supplied, locally serializes upserts matching either app name or URL, deduplicates both the app and board tile, and verifies the result by read-back. Explicit remote base URLs must use HTTPS; plaintext HTTP is accepted only on loopback. Do not put Homarr credentials in arguments, environment variables, or temporary files.

For PR delivery, use scripts/pr-deliver <pr> [--repo owner/name] [--merge]. It keeps one watcher per repository/PR, follows a changed head SHA instead of reporting stale checks, and always passes an explicit repository to gh pr merge, avoiding worktree checkout conflicts.

After pushing app changes, watch Flux converge with:

scripts/reconcile-app tools/searxng
# or
task kubernetes:reconcile-app app=tools/searxng

Do not force Flux reconciliation as the default agent workflow. Push Git changes, then watch the existing Flux controllers converge. Manual flux reconcile ... is for explicit break-glass/debug use, not normal app rollout.