feat: computed-upstream demand-driven subscription chains (piece D)

[sanity]

⚠️ PARKED — piece D (computed-upstream host-chains), the keystone

Highest-risk piece: it deletes the hollow-relay firefight and re-enables the very path #3763 disabled. REQUIRES Ian's explicit sign-off + the passing sim proofs before merge. Do NOT merge; do NOT enable auto-merge. Targets 0.2.91+, and B/E/F build on it, so it lands only after Ian's morning review. Stacked on #4657 (sim-harness); merge that first.

Problem

Hosting was holding-driven, not demand-driven, and subscription chains were hollow: a peer forwarding a SUBSCRIBE registered downstream interest but held no state and wasn't advertised as a host. Hollow chains create findability dead-ends (a GET routing toward a key lands on a non-hosting relay with nothing to serve), which the SubscribeHint placement migration papered over — and that migration became the 0.2.87 storm. The blunt anti-#3763 rule ("a relay is never a subscriber") is what created the hollow chains in the first place.

Solution

Implements the core of the (now-merged) design docs/design/demand-driven-hosting.md (§3–§6). There is no "relay" category: a peer on a live subscription chain is a real subscribed host.

• §5a interest-gated renewal — contracts_needing_renewal §1 is gated on contract_in_use (a local client OR a downstream subscriber). Un-demanded subscriptions lapse; the chain collapses inward. This is the fix: include hosted contracts in subscription renewal #3763 storm bound (renewal ∝ active demand, not accumulated cache). The eviction budget (piece A) cannot bound it: an in-use subscription is eviction-exempt AND self-renewing, so the storm set is exactly the set the budget may not touch.
• §4 computed upstream — Ring::most_keyward_hosting_neighbor derives the upstream on demand from live neighbor-hosting advertisements + ring distance (the connected host closest to the key that is strictly closer than self), never a stored formation flag. Strictly-closer makes the upstream relation a strict descent → acyclic by construction; a deterministic address tiebreak gives a total order.
• §5b/§5d renewal to the computed upstream — run_renewal_subscribe targets the computed upstream directly (first_hop), so re-rooting is automatic: when a closer host appears or the upstream drops, the next renewal follows the new computation, with no dedicated migration path. A terminus (no closer neighbor) refreshes locally; a stranded host (closer non-hosting neighbors) routes a fresh keyward subscribe to re-root.
• §3 real chain hosts — each subscribe hop becomes a real host after bubbling the reply upstream (fetch state, ring.subscribe, announce_contract_hosted), via the shared finalize_host_subscribe. Extended to piece C's terminal-advertisement consult path (a peer that subscribes via consult also hosts). The hollow-relay concept + its pin tests are removed and replaced with new-model pins.
• §6 strict-farther downstream — a downstream is registered only when the peer is strictly farther from the key than us (our upstream is excluded). This is the collapse-side dual of the strictly-closer upstream rule: together they keep the chain acyclic and its collapse terminating, closing mutual self-perpetuation (two co-hosts propping each other up forever).

Testing

Simulation proofs (the gate) — all pass across ≥3 seeds, using the sim-harness's deterministic AdvanceHostingClock + node_*_count accessors:

• test_subscription_count_tracks_demand_not_cache — no-storm: leases track active_demand_count, never hosting_count (cache); max_cycle_batch ≤ 10.
• test_subscription_chain_collapses_on_client_leave — mesh forms, then collapses to 0 survivors after disconnect + clock jump.
• test_chain_collapse_reaches_root_no_cycle — survivors == 0 (no lone survivor = no self-perpetuating lease; no pair = no mutual-upstream cycle).
• test_popular_contract_fanout_is_measured — fan-out is measured only (bounded by piece B admission, out of scope here).
• test_relay_become_host_closes_deadend — forces the relay-become-host path (chain_hosts_formed = 5–7) with a single distant SeedHostedContract holder + piece-C consult, and asserts a fresh mid-chain host holds state — a findability dead-end closed by a chain of real hosts (D's core value).

Plus unit tests: the interest-gated renewal bound (test_contracts_needing_renewal_bounded_by_active_interest adapted + a no-interest-lapses control), the new-model subscribe-driver pins, and most_keyward_hosting_neighbor.

Findings surfaced during validation

1. Relay-become-host path — in a PUT-seeded topology the contract pre-replicates to ~half the peers, so subscribes resolve at hop 1 and the relay-become-host counter reads 0; the mesh still forms (via the originator path) and the safety proofs hold regardless. test_relay_become_host_closes_deadend exercises the relay path directly with a single distant SeedHostedContract holder + consult, so D's core value is demonstrated, not just documented.
2. Demand-source divergence (cosmetic, pre-existing) — the InterestManager's local_client_count/downstream_subscriber_count can diverge from HostingManager's client_subscriptions/downstream_subscribers. Investigated + confirmed pre-existing (from feat: add delta-based state synchronization #2667) and non-blocking: every hosting/renewal/eviction/collapse decision — including this PR's gate — reads the HostingManager source (contract_in_use); the InterestManager counter only feeds broadcast-targeting + telemetry, never a hosting decision. This PR fixes only the misleading sim accessor Ring::active_demand_count to read the gate's own source. Reconciling the two managers is a separate follow-up (Reconcile drift between InterestManager LocalInterest counters and HostingManager subscription state #4660), deliberately kept out of this PR to avoid bloat/re-stale.

Refs #4642

🤖 Generated with Claude Code

https://claude.ai/code/session_01LnRCbtbjepaB5GWrb5U8SR

[AI-assisted - Claude]

[sanity]

Validation status (parked for @sanity's morning review)

All sim proofs pass, incl. §6-regression re-check. After the §6 strict-farther-downstream guard was added, I re-ran the safety proofs to confirm it didn't regress mesh formation/collapse — both pass across 3 seeds with survivors=0 (total collapse, no self-perpetuating lease, no mutual-upstream cycle):

• test_subscription_chain_collapses_on_client_leave — mesh 6/4/6 → survivors 0/0/0
• test_chain_collapse_reaches_root_no_cycle — formed_mesh 5/6/5 → survivors 0/0/0 (shape="none (total collapse)")
• test_relay_become_host_closes_deadend — chain_hosts_formed=5/7, mid-chain hosts hold state (dead-end closed)
• test_subscription_count_tracks_demand_not_cache + test_popular_contract_fanout_is_measured — pass
• Unit: most_keyward_among (computed-upstream selection: strict-closer, farther-or-equal excluded, tiebreak), interest-gated renewal bound, new-model subscribe pins.

Lint/format: piece-D code is cargo fmt + clippy clean. clippy --all-targets -- -D warnings surfaces 3 errors in files this PR does NOT touch (secrets_store/store.rs, bin/commands/update.rs, client_events/user_op_rate_limit.rs) — pre-existing base lints (clippy 1.94.0 version drift), not piece D; flagging for a base cleanup, not fixed here to keep scope tight.

Reviews: external codex review is being run and will be posted here. The adversarial Claude pass was interrupted twice by transient Anthropic API 529 Overloaded errors (recording the substitution per the multi-model-review rule); it will be re-run and posted before this leaves draft. Do NOT mark ready / merge until both reviews are posted, addressed, and you've signed off.

[sanity]

Codex review (external, non-Claude) — clean

Ran codex review --base feat/sim-harness-hosting (OpenAI codex-cli 0.141.0), which was unaffected by the Anthropic overload. Verdict:

I did not identify any discrete, actionable correctness issues in the diff.

(Codex could not run cargo in its sandbox — no network for dep resolution — so this is a diff-read review; the build/tests/clippy are verified locally, see the validation-status comment above.)

Adversarial Claude pass re-running now; will post when it lands.

[sanity]

Adversarial Claude pass — blocked by repeated Anthropic 529 overload

The adversarial Claude reviewer was re-attempted and hit a transient 529 Overloaded again (third time). It reached the high-risk-surface areas and flagged them clean before dying, but did not produce a final structured report. Recording the substitution per the multi-model-review rule.

Review posture for merge: the external non-Claude pass (codex — clean) is the load-bearing independent signal and it passed; the design was Ian-approved and merged (demand-driven-hosting.md); all 5 sim proofs + unit tests pass; codex found nothing. Before this leaves draft, please re-run the adversarial Claude pass once the overload clears (coordinator is pacing retries) so the Full-tier review is complete. No blocking findings from any pass so far.

[sanity]

Adversarial Claude review — completed, NO BLOCKERS

The adversarial pass finally completed (after three 529s). Verdict: no blockers. The acyclicity core, forward-before-contract-handling ordering, double-host prevention, and lock-safety are all confirmed sound. One MAJOR (verify) + MINORs. Full findings + my triage:

MAJOR (verify before merge) — §1 renewal now depends on HostingManager.client_subscriptions

contract_in_use reads client_subscriptions/downstream_subscribers. Pre-PR, §1 renewed all active leases; now it gates on contract_in_use, so any subscribe path that installs a lease but registers demand only in InterestManager (not add_client_subscription) would silently stop being renewed.

• Triage / partial verification: per design §5a the gate is correct (a demand-less lease should lapse). Verified the common paths are covered: WS-client and GET-with-subscribe:true register client_subscriptions via the client_events listener path (client_events.rs:98/1194), separate from finalize's add_local_client; chain hosts have a downstream_subscriber. Residual to confirm before merge: a purely-internal run_executor_subscribe with no client/downstream — does it represent real demand that must be renewed, or is lapsing correct? Action: add a regression test asserting every legitimate client-subscribe entry point makes contract_in_use true. Not a confirmed bug; a required verification.

MINORs (my triage)

• Directed-renewal no within-cycle fallback (op_ctx_task.rs:460): if the computed upstream dead-ends, the renewal fails that cycle, self-heals in ≤30s. Accept (bounded, self-healing); optionally add a greedy fallback later.
• feat: track local client interest to enable safe subscription renewal and fast cache serving #3769 local-access excluded from §1 (hosting.rs): a local-GET-without-subscription contract gets a ≤30s update gap per ~8-min lease period (§3 re-subscribes reactively). Accept + document as deliberate (local-access is a reactive-only signal); narrow + self-healing.
• §6 early-return skips PUT of a fresh contract id gives up broadcast before subscriptions resolve → state lands locally-only, remote nodes get NotFound (and fdev reports success) #4359 flush for a closer new subscriber (subscribe.rs:658): the finalize_host_subscribe upstream-registration path also flushes toward closer peers, so not generally lost. Follow-up test that a mis-rooted subscriber doesn't strand a deferred PUT.
• Hint/fallback-routed closer previous-hop declined (subscribe.rs:653): a behavior change vs pre-PR for low-connectivity/hint cases; directed renewals are safe by construction. Accept per §6 intent.
• Tiebreak doc "addressless peers sort last" is inaccurate (None sorts first; currently dead because the caller pre-filters on location()). Will fix the comment (trivial).
• Distance PartialEq(epsilon) vs Ord(exact) inconsistency (location.rs, pre-existing): not triggered here; flag for a base cleanup.

INFO

• Per-hop body-fetch/storage amplification is by design (piece D), bounded by interest-gated collapse — noted for capacity planning.

Merge posture: codex (external) clean + adversarial clean-of-blockers. The MAJOR verification + its regression test, and the two follow-up tests, should land before this leaves draft. @sanity to sign off.

[sanity]

Review findings addressed

MAJOR (resolved — verified NOT a false-lapse bug, 4900a898). The §1 renewal gate depends on HostingManager.client_subscriptions/downstream_subscribers. Traced every subscribe entry point:

• WS-client SUBSCRIBE and GET-with-subscribe:true both register client_subscriptions via the RegisterSubscriberListenerResponse handler (client_events.rs:98/:1194 → ring.add_client_subscription) — exactly the map contract_in_use reads. So a genuinely-subscribed contract is always renewed. ✅
• A chain host renews via its downstream subscriber. ✅
• run_executor_subscribe installs a lease but registers interest only in the InterestManager (add_local_client), not client_subscriptions. With no accompanying WS client / downstream (a PUT/seed with no ongoing interest), the lease correctly lapses — per design §1 a PUT only seeds and is not demand; idle contracts evaporate. This is intended piece-D behavior, not a false-lapse. The executor subscribe's real demand, if any, is the WS client that PUT/subscribed, tracked separately in client_subscriptions.

New regression test test_active_lease_renewed_iff_contract_in_use pins all four cases (seed lapses; client-sub renews; downstream renews; client-sub removal → lapse).

Doc fix (done, 4900a898). Corrected the most_keyward_among tiebreak comment (addressless peers are pre-filtered by the caller via location()? and never reach the helper).

MINORs (tracked — #4662). The three self-healing edges (§6 skips #4359 flush for a closer subscriber; directed-renewal within-cycle fallback; #3769 local-access §1/§3 gap) plus the pre-existing Distance Eq/Ord inconsistency are filed as #4662 rather than bloating this PR (one logical change). None are correctness bugs.

Review complete: codex (external) clean + adversarial Claude no-blockers + the one MAJOR verified/pinned. Ready for @sanity's sign-off.

[sanity]

Chain-host state-acquisition hardening (05ff1df) + relay-caching-masking note

Piece E's relay-caching removal surfaced that the piece-D sim proofs were passing on relay-caching contamination: with relay-caching present the contract body was already local (has_contract hit instantly), so finalize_host_subscribe's one-shot best-effort 2s fetch always succeeded. Without relay-caching a just-subscribed chain host is genuinely body-less, and the deferred-announce "UPDATE will fill the gap" fallback is broken (an UPDATE to a body-less node fails "missing contract").

Hardening added: on inline-fetch timeout, now spawns a bounded background retry that keeps pulling the body (greedy GET + terminal-advertisement consult, robust to a still-forming body-less responder), then announces once it lands — the §5d proximity/summary-sync, deferring announce/serve until synced (invariant 1). Bounded by body-present / demand-fade / attempt budget. It is a no-op with relay-caching present (inline fetch still succeeds), so it does not change this branch's passing proofs; it is the correctness prerequisite for E.

Not yet resolved (reported to Ian for direction): validating with relay-caching REMOVED (cherry-picked E's PUT-removal onto a throwaway branch), the state-acquisition fix is necessary but not sufficient — 3/5 proofs still fail because the dominant issue is subscribe reachability, not body-landing: with 8 concurrent subscribers and a single holder, ~6/8 subscribes dead-end (NotFound, no lease; chain_hosts_formed=1, max_upstreams=0; seed-dependent — some seeds form a mesh of 5). Removing relay-caching drops host/advertisement density, and subscribe routing + neighbor-scoped consult can't reliably bridge far subscribers to the single holder. That is a broader convergence/mesh-bootstrap question than the scoped state-acquisition fix. See report to coordinator.