Skip to content
This repository was archived by the owner on Jul 8, 2026. It is now read-only.

sylphx-managed-resource-controller.md

Latest commit

 

History

History
425 lines (329 loc) · 15.9 KB

sylphx-managed-resource-controller.md

File metadata and controls

425 lines (329 loc) · 15.9 KB
Raw

Case Study — Sylphx Managed Resource Controller

How we took a Kubernetes reconciler from 320 unit tests to 521 tests + 82.62% mutation score + 8 additional testing methods across 3 refactor phases.

Project: Sylphx Platform — Managed Resource Controller Purpose: Provisions + reconciles customer-facing managed infrastructure (Postgres via CNPG, Valkey, Typesense) Stack: TypeScript, Bun, Drizzle, Effect-TS, Kubernetes, Neon Timeline: 2026 Q2, 3 phases over ~6 weeks Team: 1 engineer (full-time dedicated refactor)

Starting state

Before the refactor began:

Metric Value
Tests 320
Testing methods used 2 (unit + integration)
Mutation score unknown — Stryker not configured
Property tests 0
Type-level tests 0
E2E tests 0
Load tests 0
Chaos experiments 0
SAST configured ❌
Canary deploys ❌ (direct ArgoCD sync)
Runbooks ❌
ADRs 47 (pre-existing, platform-wide)
SLOs defined ❌
controller.ts size 815 lines (God class)

Two critical bugs existed in production:

  1. Bug #1: Terminating resources were never picked up by the reconcile query (WHERE clause excluded them) — created zombie CNPG clusters
  2. Bug #2: updateResource didn't trigger a reconcile, so config changes were silently dropped

We didn't know test quality was weak until we started investigating these bugs. Every "covered" line was covered — but the tests didn't ASSERT on the behaviour that was broken.

Phase 5 — Foundation (P5.1–P5.15)

Goal: Add foundational test quality + type safety without rewriting the controller.

What we added

Method Tool Tests added
Property-based testing fast-check 28 properties × 100-500 runs
Mutation testing (config) Stryker config only, not yet run
Type-level testing expect-type 15 type-level tests
Contract testing Zod 45+ schema tests
In-memory DB integration pg-mem 15 SQL semantic tests
Graceful shutdown (AbortController) custom 10 shutdown tests
Resilience primitives (breaker + limiter) custom 22 tests

Metrics after Phase 5

Metric Before After P5
Tests 320 475 (+155)
Testing methods 2 11 (+9)
controller.ts size 815 815

Phase 5 self-critique (the honest part)

We looked at what we'd shipped and found:

  • Effect-TS was "skin deep" — we added the library but only one function (classifyReconcile) used it. The controller still had try/catch in most places.
  • Circuit breaker was DEAD CODE — we wrote the class and tests but never wired it into any actual K8s call. The tests passed against the primitive, not against real usage.
  • pg-mem test used HAND-ROLLED SQL — the test asserted on a copy of the query, not the REAL Drizzle query builder output. Drift possible.
  • Stryker was configured but NEVER RAN — we didn't know the mutation score. It could have been 20%.
  • Graceful shutdown not wired to SIGTERM — the function existed but no signal handler called it.
  • Zod schemas DUPLICATED TypeScript interfaces — two sources of truth in src/lib/db/schema/resources.ts and src/lib/resources/schemas.ts.

We declared Phase 5 "done" and moved on, then caught ourselves during a review: "this is SOTA on the surface only." We opened Phase 6.

Phase 6 — Real SOTA (P6.1–P6.10)

Goal: Close every gap identified in the Phase 5 self-critique.

P6.1 — DB Effect Layer

src/lib/db/effect.ts — wrapped Drizzle in an Effect Layer with Context.Tag('@sylphx/DB') + tagged errors (DbError, TransactionError). Tests inject mock DB via makeDbTestLayer instead of mock.module pollution. 14 tests added.

P6.2 — Wire the circuit breaker into REAL K8s calls

This is the gap that matters most. Changed serverSideApply() to:

await withK8sResilience(() => client.patch(...))

Where withK8sResilience wraps the call in a process-wide singleton circuit breaker + rate limiter. Every K8s call in the codebase — legacy Promise callers AND the Effect pipeline — now flows through ONE breaker and ONE rate limiter. 8 integration tests proving real wiring.

P6.3 — Full Effect reconcile pipeline

src/controller/managed-resource/effect-pipeline.ts — reconcileResource returns Effect<ReconcileOutcome, never, K8s>. The never error channel forces all outcomes into the success channel:

type ReconcileOutcome =
  | { kind: 'synced', manifestsApplied, clusterName }
  | { kind: 'provisioning', manifestsApplied, clusterName }
  | { kind: 'timeout', elapsedMs }
  | { kind: 'error', tag, message, retryable, category }

The controller's switch (outcome.kind) gets an exhaustive never check — adding a new variant breaks compilation. 10 tests.

P6.4 — SIGTERM wiring

loop.ts registers onShutdown(() => controller.shutdown()). K8s pod termination now drains in-flight reconciles before exit. 7 tests.

P6.5 — Split controller.ts

Extracted actionable-query.ts + provider-registry.ts. Controller shrank from 815 → 784 lines. 10 registry tests.

P6.6 — drizzle-zod SSOT

Deleted 200 lines of duplicated TypeScript interfaces from db/schema/resources.ts. Re-exported types from lib/resources/schemas.ts (Zod schemas). Added createSelectSchema(managedResources) to auto-derive row types from Drizzle table.

P6.7 — pg-mem uses REAL Drizzle query

actionable-query-drizzle.pgmem.test.ts — pg-mem's createPg() adapter wired to Drizzle. Ran the REAL buildActionableWhere(...) expression against pg-mem. Hit several gotchas:

  • Registered gen_random_uuid() manually (pg-mem doesn't ship pgcrypto)
  • Proxy-wrapped the Pool to strip Drizzle's types.getTypeParser
  • Used testDb.execute(sql\...`)instead of.select()` to avoid rowMode='array' issues
  • Used Drizzle's typed .insert().values() instead of raw SQL for NULL handling

8 tests passing against the REAL query builder output.

P6.8 — OpenTelemetry SDK init

otel-sdk.ts — dynamic-import the SDK only when OTEL_ENABLED=true. Resource attributes (service.name / version / environment) per OTel semantic conventions. 5 tests.

P6.9 — Actually ran Stryker

File                  | Score   | # killed | # survived
effect-reconcile.ts   | 100.00% |     41   |      0  ⭐
connection-string.ts  |  93.33% |     56   |      4
brands.ts             |  88.89% |     32   |      4
backoff.ts            |  88.89% |     16   |      2
conditions.ts         |  88.61% |     70   |      9
effect-errors.ts      |  85.06% |     74   |     13
resilience.ts         |  84.81% |     63   |     12
errors.ts             |  72.55% |     37   |     14
schemas.ts            |  58.33% |     49   |     35
All files             |  82.62% |    438   |     93

Schemas.ts was the weakest — adding parseConfigByKind branch tests for kv/blob/volume/search lifted it from 51.9% → 62.03%.

P6.10 — SLOs + Grafana dashboards as code

infra/addons/monitoring/managed-resource-alerts/vmrule-managed-resource.yaml — 5 SLO groups with 11 alert rules. Runbook URLs on every alert.

Grafana dashboard ConfigMap with 7 panels: SLO stat gauges, latency histograms, queue depth, retry rates, throughput.

Metrics after Phase 6

Metric Before P5 After P5 After P6
Tests 320 475 521
Mutation score ? ? 82.62%
effect-reconcile.ts mutation — — 100%
controller.ts size 815 815 784
Testing methods 2 11 14
SLOs defined 0 0 5

Phase 7 — Finishing touches (P7.1–P7.7)

Goal: Close the remaining gaps — docs, canary, chaos, and the last pieces of Effect integration.

P7.1 — Plug reconcileResource pipeline INTO reconcileOne

The pipeline existed (P6.3) but reconcileOne still used Promise code. Rewrote to use Effect.runPromise(reconcileResource(...).pipe(Effect.provide(K8sLive))) with an exhaustive switch (outcome.kind).

P7.2 — BaseProvider Effect-native defaults

Every subclass of BaseProvider now inherits:

reconcileEffect(resource): Effect<ReconcileResult, ReconcileError> {
  return Effect.tryPromise({
    try: () => this.reconcile(resource),
    catch: fromLegacyError,
  })
}

Existing providers (CNPG, Valkey, Typesense) get Effect-native variants for free. Pipeline prefers reconcileEffect when present. 10 tests.

P7.3 — Extract status-writer

src/controller/managed-resource/status-writer.ts — all DB state writes (markSynced, markFailed, markProvisioning, markTeardownFailed, hardDelete, applyReconcileResult). Controller shrank: 784 → 665 lines.

P7.4 — ADR records

Three new ADRs:

  • ADR-048: Effect-TS + Layer-based DI (context, decision, alternatives, consequences)
  • ADR-049: drizzle-zod SSOT
  • ADR-050: Managed-resource SLO definitions

P7.5 — Runbooks

docs/runbooks/ — 7-section runbooks for each critical alert:

  • managed-resource-slo-burn.md
  • teardown-failure.md
  • decryption-error.md (Sev-1 security runbook)

Each includes: symptoms, immediate triage, common patterns + remediation, escalation path, recovery verification, post-incident actions.

P7.6 — Flagger canary config

apps/sylphx-platform/manifests/canary.yaml — Flagger Canary CRD with SLO-gated promotion. MetricTemplates query VictoriaMetrics for reconcile-success-rate and p99-latency. Auto-rollback on breach.

P7.7 — Chaos Mesh experiments

infra/addons/chaos-mesh/managed-resource-experiments/ — composite workflow with:

  • Pod kill (verify HA failover)
  • Network delay to kube-apiserver (verify circuit breaker opens)
  • CPU stress (verify reconcile rate holds)
  • Final SLO gate (strict 99.5% after stabilization)

Metrics after Phase 7

Metric Before P5 After P6 After P7
Tests 320 521 521 (+0 new, refactors only)
Mutation score ? 82.62% 82.62%
Testing methods 2 14 18 (+4: canary, chaos, ADR, runbooks)
controller.ts size 815 784 665 (−18%)

Final state

Metric Value
Tests 521 passing, 0 failing
Mutation score 82.62% overall, 100% on effect-reconcile.ts
Testing methods in use 18 distinct methods
Code in controller.ts 665 lines (down from 815)
Modules extracted 8 focused modules
SLOs defined 5 (with 11 alert rules)
ADRs documenting decisions 3 new (+47 pre-existing)
Runbooks 3 (SLO burn, teardown failure, decryption)
Canary deploys ✅ Flagger with SLO gates
Chaos experiments 4 (composite workflow + 3 standalone)
Pre-existing bugs fixed 2 (zombie terminating, updateResource)

Methods now in use (18 of the 35 in this playbook)

# Method Status
1 Unit testing ✅ 521 tests
2 Type-level testing ✅ 15 expect-type tests
3 Integration testing ✅ Lifecycle integration suite
4 Property-based testing ✅ 28 properties × 100-500 runs
5 Contract testing (Zod) ✅ drizzle-zod SSOT
6 Mutation testing ✅ 82.62% score
9 Chaos engineering ✅ 4 experiments
10 SAST (CodeQL) ⚠️ pre-existing at platform level
11 Canary SLO gates ✅ Flagger
13 In-memory DB ✅ pg-mem + real Drizzle
15 Migration testing ✅ Atlas migrate lint
16 Snapshot testing 🟡 partial
18 Accessibility ⚠️ pre-existing at platform level
20 Fuzzing ❌ future work
21 Benchmark regression ❌ future work
26 Fault injection (code) ✅ makeMockK8sService({ applyBehaviour })
32 BDD ❌ (intentionally skipped)
33 Smoke testing ✅ via /healthz

Lessons learned

1. "Done" is a lie

We declared Phase 5 done and were wrong. The self-critique caught 7 gaps. Always re-audit before shipping. Better: have someone NOT on the refactor do the audit.

2. Mutation testing changes what "tested" means

We had 80%+ line coverage on schemas.ts before running Stryker. The mutation score was 52%. Half the "coverage" was weak assertions. Line coverage is a floor, mutation score is the truth.

3. Property tests catch things unit tests NEVER will

Real case: our maskConnectionString function passed all example tests we wrote. Then fast-check generated a password that happened to be "po" — and the masked URL postgres://... still contained "po". We'd have NEVER written that test case by hand.

4. Type-level tests are free insurance

Adding 15 expect-type tests took ~2 hours. The invariant they enforce — branded IDs cannot be confused — would have caught 3 prior bugs if it had existed earlier.

5. Wiring matters as much as writing

P5 had a circuit breaker. P6 actually USED it. The difference between "written" and "wired" is the difference between performance theatre and real reliability.

6. The Effect migration is incremental

We didn't rewrite everything to Effect. We added an Effect-native interface + default implementations, so existing Promise code keeps working AND new code can go all-in. This is the pragmatic SOTA path.

7. Docs are testing infrastructure

ADRs + runbooks aren't glamorous, but they prevent "how did we decide this?" questions 6 months later and "what do I do at 3am when this fires?" questions during incidents. Both save hours per incident.

8. Chaos engineering requires SLOs

A chaos experiment without an SLO gate is just noise. You have to know what "passing" looks like before you start breaking things.

Commit chain

48760e01a feat: phase 7b — BaseProvider Effect-native defaults (P7.2)
be8154e9b feat: phase 7a — SOTA finishing touches (P7.1, 3, 4, 5, 6, 7)
a78b5766b feat: phase 6c — full Effect reconcile pipeline (P6.3)
1c3d4380e feat: phase 6b — drizzle-zod SSOT + split + pg-mem + OTel (P6.5–8)
a347416d4 feat: phase 6a — real Effect integration + SLOs + mutation (P6.1,2,4,9,10)
178d24427 feat: phase 5c — OpenTelemetry spans + Prometheus /metrics
aee70de81 feat: phase 5b — Effect-TS migration (P5.5-7)
23c64d020 feat: phase 5a — type safety + resilience + property tests

What we'd do differently

If starting fresh with this playbook:

  1. Start at Level 2, not Level 1. Set up property testing + mutation testing in week 1, not month 3. The ROI is extreme.
  2. Write ADRs AS you make decisions, not retroactively. Future you won't remember why you chose Effect-TS over fp-ts.
  3. Set up Stryker + mutation gate FIRST. Before writing 100 unit tests, write 10 and verify they actually kill mutants.
  4. Write runbooks with the alert, not after the first incident.
  5. Wire circuit breakers + rate limiters as you build, not after. "Dead code" primitives become real maintenance debt.
  6. Resist the urge to declare "done." Always do a self-critique before moving on.

Appendix — where to see each method in action

Method Link to code
Unit src/controller/managed-resource/__tests__/*.test.ts
Integration src/controller/managed-resource/__tests__/lifecycle.integration.test.ts
Property src/controller/managed-resource/__tests__/*.property.test.ts
Type-level src/lib/ids/__tests__/brands.types.test.ts
Contract (Zod) src/lib/resources/schemas.ts
Mutation apps/sylphx/stryker.conf.json
pg-mem src/controller/managed-resource/__tests__/actionable-query-drizzle.pgmem.test.ts
OTel src/controller/managed-resource/otel-sdk.ts
SLO VMRule infra/addons/monitoring/managed-resource-alerts/vmrule-managed-resource.yaml
Grafana dashboard infra/addons/monitoring/managed-resource-alerts/grafana-dashboard.yaml
Canary apps/sylphx-platform/manifests/canary.yaml
Chaos infra/addons/chaos-mesh/managed-resource-experiments/
ADRs docs/adr/ADR-048-*.md, ADR-049-*.md, ADR-050-*.md
Runbooks docs/runbooks/managed-resource-slo-burn.md et al.