Skip to content

Commit 0603841

Browse files
philcunliffeclaude
andauthored
Remote config and join flow (LLP 0025) (#98)
* Add LLP 0022: remote config and join flow spec Specifies the client half of centrally-managed gateway configuration: join sequence, config pull loop, seed-config mode, staged-restart apply semantics, hash-pinned install-on-config, and last-known-good rollback with post-apply probation. Supporting amendments: - LLP 0003: config apply engine added to the core-owns list - LLP 0011: join added as a non-interactive entry point - LLP 0017: staged restart for config replacement + installer relaunch requirement - central proto.md: policy tokens, running-config If-None-Match convergence semantics, hash-pinned plugins, 404 demoted to legacy - LLP 0000: subsystem map updated - notes-archive: round-1 review record for LLP 0022 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Renumber remote-config spec to LLP 0023 Origin master took 0022 for iceberg-export-partitioning (#91) while the join-flow spec was drafted on a local branch; renumber to the next free slot and update all cross-references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Implement remote config and join flow (LLP 0023) Kernel side: - Config apply engine (src/core/config/apply.js): validate -> install pinned plugins -> persist to A/B slots -> atomic pointer flip -> staged restart, with last-known-good rollback, bad-etag re-apply backoff, and structured rollback reasons. The served etag lives in a per-slot sidecar written before the flip, so the document and its etag commit on the same rename in both directions. - Narrow ctx.configControl facade (stage / confirmPoll / runningEtag) exposed only in daemon mode; CLI boots leave it undefined so transport plugins keep their pull loops off. - Kernel-owned probation watchdog: window max(3 x poll_interval, 120s) sized from the staged document, cleared only by a confirmed poll, expiry also evaluated at boot before plugin activation (crashloop case), orphaned markers from a crash before the flip are discarded. - Staged restart: DAEMON_RESTART_EXIT_CODE (75) for foreground invokers; service managers already relaunch via KeepAlive / Restart=always (now pinned by tests). Sinks are closed on shutdown so plugin timers stop. - Hash-pinned install-on-config through the LLP 0007 path; pin verified against the staged artifact before the install commits. Bundled first-party plugins: version strict, hash skipped. - Config shape: plugin entries accept version / artifact_hash / source pins. - hypaware status: remote-config section (probation, last rollback + reason, remembered bad etag, running etag) in text and JSON. - hypaware join <url> [token] [--token-file|stdin] [--no-daemon]: writes the mode-0600 seed config and runs the daemon install — a wrapper over the two existing steps, not a second code path. Central plugin (transport only): - Config pull loop (central/src/config_client.js): immediate pull on bootstrap success, steady timer, If-None-Match always the running config's etag, 401 refresh-retry, 404 legacy backoff, 429/503 Retry-After + linear backoff, 1 MiB body cap. - Dropped the never-wired config_etag_path option (the etag is kernel-managed and read through the facade). Settled LLP 0023 open questions: poll default 300s, probation floor 120s, max document size 1 MiB. proto.md sidecar wording updated; the restart exit code recorded in LLP 0017. Tests: apply-engine state machine (18), pull loop (10), join command (5), installer relaunch defaults; join_flow_remote_config hermetic smoke drives seed -> bootstrap -> pull -> apply -> restart -> probation clear against a stub server with convergence assertions. Note: central_forward_outbox was already failing on origin/master (empty ingest rows) before this change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Flip LLP 0023 to Active Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address dual-review findings on remote config (PR #98) Apply engine (LLP 0023): - Reorder stage() to shape-check -> install pinned plugins -> full validation, so a served config can name a not-yet-installed plugin (catalog-backed validation only knows a plugin once it is installed). LLP 0023 install-on-config section updated with the ordering rationale. Config pull transport: - Enforce the 1 MiB cap before buffering: oversized Content-Length is rejected without reading, chunked bodies stream through a byte counter that cancels at the cap. - Per-poll AbortController with a 30s request deadline covering request and body read; stop() aborts an in-flight poll after a 1s drain grace so a stalled config GET cannot wedge daemon shutdown. Tests: - New test/core/config-apply-deps.test.js covering the real pin enforcement: bundled version mismatch, bundled hash exemption, satisfied-lock skip, artifact hash mismatch via a local git fixture, and install-then-validate over a fresh catalog. - Pull-loop tests for Content-Length pre-reject, chunked cap cancel, stop() abort, request deadline, 429/503 Retry-After, and parseRetryAfter (now exported). - Engine ordering test (install before validate) and shape-gate test. Typecheck and style: - Fix broken inline type-import path and literal exit-code comparisons that broke npm run typecheck; replace remaining inline import() types with top-of-file @import blocks. - Reword closeAllSinks JSDoc: identity refresh is lazy and has no timer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Don't unref the poll deadline and stop-grace timers An unref'd timer can't fire once a wedged fetch is the only live handle: the event loop drains, so the deadline/grace abort never happens — in CI this surfaced as node:test cancelling the pull-loop tests with 'Promise resolution is still pending but the event loop has already resolved'. Both timers are cleared as soon as the poll settles, and the loop's policy is no-unref anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Renumber remote-config spec to LLP 0025 (0024 is claimed by open PR #99) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 322aaed commit 0603841

31 files changed

Lines changed: 3989 additions & 35 deletions

collectivus-plugin-kernel-types.d.ts

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -323,10 +323,60 @@ export interface PluginActivationContext {
323323
* before appending to the cache.
324324
*/
325325
backfillMaterializers: BackfillMaterializerRegistry
326+
/**
327+
* Narrow facade over the kernel config apply engine (LLP 0023). Only
328+
* present when the host process runs an apply engine (the daemon);
329+
* absent in plain CLI boots, so transport plugins must treat it as
330+
* optional and skip their pull loops when it is missing. The facade
331+
* is the only channel a plugin has into config application — the
332+
* kernel owns validation, install, persistence, restart, probation,
333+
* and rollback.
334+
*/
335+
configControl?: ConfigControlFacade
326336
requireCapability<T = unknown>(name: CapabilityName, range?: SemverRange): T
327337
provideCapability<T = unknown>(name: CapabilityName, version: SemverVersion, value: T): void
328338
}
329339

340+
/**
341+
* Plugin-facing surface of the kernel config apply engine. Handed to
342+
* transport plugins (e.g. `@hypaware/central`) so they can deliver a
343+
* downloaded config document and report poll liveness. Deliberately
344+
* narrow: plugins never see probation state, slot paths, or rollback
345+
* bookkeeping.
346+
*/
347+
export interface ConfigControlFacade {
348+
/**
349+
* Deliver a downloaded config document (parsed JSON) plus the ETag it
350+
* was served under. The kernel validates, installs pinned plugins,
351+
* persists, swaps, and requests a staged restart. Resolves before the
352+
* restart happens; callers should treat `{ ok: true }` as "apply
353+
* committed, restart pending".
354+
*/
355+
stage(document: unknown, etag: string): Promise<ConfigStageResult>
356+
/**
357+
* Report a successful authenticated config poll (200 or 304). Clears
358+
* the post-apply probation window when one is active; a no-op
359+
* otherwise.
360+
*/
361+
confirmPoll(): void
362+
/** ETag of the *running* config, for `If-None-Match`. Undefined when the operative config was never applied from the server (e.g. seed). */
363+
runningEtag(): string | undefined
364+
}
365+
366+
export type ConfigStageResult =
367+
| { ok: true, action: 'applied' | 'noop_same_etag' | 'skipped_bad_etag' }
368+
| { ok: false, errorKind: ConfigApplyErrorKind, message: string }
369+
370+
export type ConfigApplyErrorKind =
371+
| 'config_invalid'
372+
| 'plugin_install_failed'
373+
| 'artifact_hash_mismatch'
374+
| 'bundled_version_mismatch'
375+
| 'document_too_large'
376+
| 'apply_engine_not_ready'
377+
| 'restart_pending'
378+
| 'apply_io_error'
379+
330380
export interface PluginDeactivationContext {
331381
plugin: ActivePlugin
332382
log: PluginLogger
@@ -405,6 +455,26 @@ export interface PluginConfigInstance {
405455
name: PluginName
406456
enabled?: boolean
407457
config?: JsonObject
458+
/**
459+
* Pinned plugin version. Set by centrally-served configs (LLP 0023):
460+
* the apply engine refuses a config whose pins it cannot satisfy.
461+
* For bundled first-party plugins the pin is checked strictly against
462+
* the bundled version; for fetched plugins it selects the artifact.
463+
*/
464+
version?: SemverVersion
465+
/**
466+
* Pinned artifact content hash for fetched plugins. The apply engine
467+
* verifies the fetched artifact against this before committing the
468+
* install; a mismatch is an apply failure. Ignored (not checked) for
469+
* plugins bundled with the running kernel.
470+
*/
471+
artifact_hash?: string
472+
/**
473+
* Optional explicit install source (raw source string accepted by the
474+
* plugin installer). Defaults to the plugin name, which the resolver
475+
* maps to its canonical git source.
476+
*/
477+
source?: string
408478
}
409479

410480
/**

hypaware-core/plugins-workspace/central/index.js

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
import path from 'node:path'
44

55
import { validateCentralConfig } from './src/config.js'
6+
import { createConfigPullLoop } from './src/config_client.js'
67
import { IdentityClient } from './src/identity_client.js'
78
import { createForwardSink } from './src/sink.js'
89

@@ -26,6 +27,10 @@ import { createForwardSink } from './src/sink.js'
2627
export async function activate(ctx) {
2728
const query = ctx.query
2829
const storage = ctx.storage
30+
// Present only in daemon mode. Without an apply engine there is no
31+
// one to hand a pulled document to, so the pull loop stays off (CLI
32+
// boots must not fire config polls as a side effect of `hyp status`).
33+
const configControl = ctx.configControl
2934

3035
ctx.sinks.register({
3136
name: 'forward',
@@ -55,13 +60,35 @@ export async function activate(ctx) {
5560
hyp_identity_source: source,
5661
})
5762

58-
return createForwardSink({
63+
const sink = createForwardSink({
5964
config,
6065
identityClient,
6166
query,
6267
storage,
6368
log: sinkCtx.log,
6469
})
70+
71+
if (!configControl) return sink
72+
73+
// @ref LLP 0025#config-pull-loop [implements] — pull immediately on bootstrap success, then on the steady timer
74+
const pullLoop = createConfigPullLoop({
75+
centralUrl: config.url,
76+
identityClient,
77+
configControl,
78+
...(config.poll_interval_seconds !== undefined
79+
? { pollIntervalSeconds: config.poll_interval_seconds }
80+
: {}),
81+
log: sinkCtx.log,
82+
})
83+
pullLoop.start()
84+
85+
return {
86+
...sink,
87+
async close() {
88+
await pullLoop.stop()
89+
await sink.close()
90+
},
91+
}
6592
},
6693
})
6794
}

hypaware-core/plugins-workspace/central/proto.md

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -30,8 +30,9 @@ atomic tmp+rename).
3030
### POST `/v1/identity/bootstrap`
3131

3232
Exchange an operator-issued bootstrap token for a long-lived JWT.
33-
Bootstrap tokens are single-use; a successful bootstrap response also
34-
invalidates the bootstrap token server-side.
33+
Bootstrap tokens are **policy tokens** (server LLP 0008): multi-use, so
34+
one token can be deployed fleet-wide via MDM, and every token references
35+
a config at mint (see "Config pull" below).
3536

3637
Request:
3738

@@ -78,6 +79,12 @@ Headers (request):
7879
- `Authorization: Bearer <jwt>`
7980
- `If-None-Match: <etag>` (optional)
8081

82+
`If-None-Match` reflects the **running** config, never a
83+
downloaded-but-not-yet-applied one. The server reads this header to
84+
track fleet convergence, so a gateway mid-install/mid-apply keeps
85+
presenting its old etag until the new config has taken effect
86+
(LLP 0025).
87+
8188
Response 200:
8289

8390
```json
@@ -89,15 +96,23 @@ Response 200:
8996
}
9097
```
9198

99+
The body is a full HypAware v2 config and replaces the gateway's
100+
operative config wholesale. Plugin entries are pinned by **version +
101+
artifact content hash**; the gateway verifies the artifact hash on
102+
install and treats a mismatch as an apply failure (LLP 0025).
103+
92104
`ETag: <hex>` accompanies every 200 response. Clients persist the etag
93-
in a sidecar (`<plugin.stateDir>/config-etag.json`) so a restart
94-
short-circuits to 304 instead of re-pulling and re-validating.
105+
of the *running* config in kernel-managed state (it transitions
106+
atomically with the operative config on apply and rollback — LLP 0025)
107+
so a restart short-circuits to 304 instead of re-pulling and
108+
re-validating.
95109

96110
Response 304: no body. The gateway keeps its current config.
97111

98-
Response 404: the operator has not registered a config for this
99-
gateway. Gateways back off to 5 minutes and log once until the state
100-
clears.
112+
Response 404: legacy-only branch — every token now references a config
113+
at mint (server LLP 0009), so gateways enrolled under that flow always
114+
resolve. Kept for conformance against older servers: back off to
115+
5 minutes and log once until the state clears.
101116

102117
Response 401: see "Refresh window" above.
103118

hypaware-core/plugins-workspace/central/src/config.js

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,6 @@ export function validateCentralConfig(value) {
5858
}
5959
}
6060

61-
if (cfg.config_etag_path !== undefined && typeof cfg.config_etag_path !== 'string') {
62-
return invalid('central.config_etag_path must be a string when set')
63-
}
64-
6561
return { ok: true, config: /** @type {CentralSinkConfig} */ (/** @type {unknown} */ (cfg)) }
6662
}
6763

0 commit comments

Comments
 (0)