docs(roadmap): read-path verdict — 12.7x -> 4.0x (GO 8/8 pairs); floor = OPEN+FLUSH round trips, next levers logged

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

Author: factory-ain3sh

.agents/specs/2026-06-11-per-phase-1-5x-roadmap-read-ttls-per-request-cost-native-bulk-ingest.md

@@ -13,7 +13,7 @@
 | diff | 2.79x | ≤1.5x | WS1+WS2 |
 | edit | 14.5x (8ms) | ≤3ms absolute; 1.5x likely unreachable at this absolute scale, recorded honestly | WS2 |
 | fsck | 1.07x | hold ≤1.5x | — |
-| read-path warm steady | 12.7x | ≤1.5x | WS1 |
+| read-path warm steady | **4.0x** (was 12.7x; WS4 flush-inval fix + dir cache) | ≤1.5x missed; floor = open+flush FUSE round trips per open/close cycle | WS4 done; next lever = keep-cache for upper/DB layers |
 
 First commit: write this scoreboard + plan to `.agents/specs/2026-06-11-per-phase-1.5x-roadmap.md` and update it after each workstream's verdict.
 
@@ -50,6 +50,7 @@ Kill-switch-gated implementation → SDK/CLI tests + clippy/fmt → correctness
 Order: WS1 → WS2 → WS3, re-running the full scoreboard after each so the artifact always reflects measured reality.
 
 ## Status log
+- **WS4 / read-path per-request (2026-06-11): DONE — warm steady-state 12.7x → ~4.0x (GO, 8/8 pairs, paired wall median 0.744); ≤1.5x missed, floor identified.** Root cause found by stepping the keep-cache state machine against counters: the FLUSH handler invalidated the inode unconditionally, so every close(2) of a READ-ONLY fd permanently revoked `FOPEN_KEEP_CACHE` eligibility (the drift guard's `dropped` set is sticky) — 64 grants vs 1,216 stale rejections on the read profile; every re-open of an unchanged base file paid a fresh FUSE READ. Fix: FLUSH only invalidates when it actually moved buffered writes (kill switch `AGENTFS_FUSE_FLUSH_INVAL=1`); per-WRITE invalidation already covers threshold-drained buffers. Counters after: keep-cache granted 1,280/1,280, READs 1,280→64, stale rejections 0. Two more levers landed: `opendir` now grants `FOPEN_CACHE_DIR|FOPEN_KEEP_CACHE` (requires dropping `FUSE_NO_OPENDIR_SUPPORT`; readdirplus 482→24 on the read profile, 2,858→1,425 on the git workload; kill switch `AGENTFS_FUSE_CACHE_DIR=0`) and open() collapsed from 3 `block_on` hops to 1. Git workload (deterministic counters; wall too noisy on today's loaded host): total dispatches −7.9% (64.8k→59.7k), getattr −2.2k, invalidations 21.5k→15.2k; status phase 6.33x→1.99x median across 4 pairs. Correctness: phase8 suite green (only the two pre-existing stale perf-threshold gates fail; repeated-read gate itself improved to 3.0x), metadata-mutation + writeback-durability green, workload digests equivalent in all 16 A/B runs. Residual floor: each open/read/close cycle still pays the OPEN+FLUSH synchronous FUSE round-trip pair (~60µs vs native ~14µs) — ≤1.5x is unreachable for open/close-bound shapes through FUSE. Next levers logged in notes: extend `keep_cache_for_read_open` beyond `Layer::Base` to upper/DB-backed files (requires relaxing the drift guard's sticky drop to fingerprint revalidation), and FUSE passthrough for read fds.
 - **WS3 (2026-06-11): DONE — `agentfs clone` lands at 2.34x (from 8.41x; target ≤1.5x missed, recorded honestly).** SDK `AgentFS::import_entries` bulk import (bounded multi-inode transactions, parents-before-children, inline/chunked/symlink storage, dentry UNIQUE → AlreadyExists) + CLI `agentfs clone <db> <source> [name]`. Pipeline deviates from spec (see notes): `git clone --no-checkout` through a temp mount → `ls-tree -r -z` + `cat-file --batch` → `import_entries` → fabricate git index v2 with cached stat data matching what the FS serves (ino/dev/size/times/sha), instead of `git archive | import` + `update-index --refresh` (refresh would re-stat+re-read every file through FUSE). Acceptance benchmark (`scripts/validation/agentfs-clone-benchmark.py`, codex fixture, 5 iters): native median 0.374s, agentfs 0.875s, ratio 2.34x (paired 2.48x), every iteration verified — `git status` clean through a FRESH mount, `git fsck --strict` clean, sha256 worktree hash identical to native. Stage budget (`AGENTFS_CLONE_TIMINGS=1`): git-clone-no-checkout 330ms (pack write into DB), import 288ms (42.8MB → DB), cat-file 104ms, ls-tree 37ms, index 6ms, process+mount ~85ms. Residual gap is the content double write (pack + worktree, both into the single DB — same shape as native's pack+worktree but against SQLite txns); candidate future shaves: overlap cat-file with import, larger import txns, shared-clone pack reuse. Limitations: no submodules, no smudge/clean filters, SHA-1 repos only.
 - **WS2 (2026-06-11): DONE (instrumentation + create fast path + critical-path discovery; deep per-request work deferred behind WS3).** Per-op dispatch latency counters added (`fuse_op_<op>_{count,nanos}`, dispatch-wrapped parse→handler→reply). Findings: dispatch-time ranking ≠ critical-path ranking — setattr (857ms-1.2s) is issued async by kernel writeback and never blocks git (deferred-SETATTR A/B parity re-confirmed at today's HEAD, paired median 1.008 → stays opt-in permanently). Git-visible sync ops in clone ≈ 1.07s of the 2.84s overhead; the rest is queue wait, kernel round trips, and SQLite write-lock contention (sync creates queue behind async setattr txns). create_file fast path: existence pre-check SELECT replaced by dentry UNIQUE-constraint mapping, parent mtime/ctime stashed into the batcher overlay instead of an in-txn UPDATE → 145µs → 125µs (txn-boundary ~115µs floor now dominates; only create-deferral or WS3 bypass goes lower). Conclusion: FUSE clone bottoms out ~5x even with all sync dispatch zeroed → WS3 `agentfs clone` is the only ≤1.5x clone route; read-path per-request work (read 83µs, open 46µs) revisited after WS3.
 - **WS1 (2026-06-11): DONE, minor lever.** Entry/attr TTL default 1s→10s (neg stays 1s). Git workload: lookups −32% (18.2k→12.3k), getattrs +2.6k (revalidation shift), net dispatches −4-9%; wall time flat. Read-path steady-state hypothesis falsified: request counts identical across TTLs (one round trip per object per mount); its ≤1.5x target moves to WS2 (per-request cost, measured ~98µs/req on metadata-heavy paths). Cross-mount sanity passed (create ≤1s, modify immediate; `run --session` joins the same mount). Correctness gates green; phase8 perf thresholds pre-existing stale (followup logged).

.agents/specs/2026-06-11-per-phase-1-5x-roadmap-read-ttls-per-request-cost-native-bulk-ingest.notes.md

@@ -6,6 +6,21 @@ User comment: none
 
 ---
 
+## 2026-06-11T12:30-07:00 — Read-path 12.7x root cause: FLUSH on read-only fds permanently revoked keep-cache
+**Type**: surprise
+**Context**: Counters on the read profile showed `base_fast_open_keep_cache=64` vs `base_fast_open_rejected=1216` with `base_fast_inode_invalidations=1280` — one invalidation per close. Stepping the state machine: every close(2) sends FLUSH; the handler called `invalidate_inode_cache_self` unconditionally, which feeds the drift guard's STICKY `dropped` set, so the first close of a file revoked `FOPEN_KEEP_CACHE` eligibility forever. Each re-open of an unchanged base file then re-read everything through FUSE. The kernel page cache was being destroyed by the very flag machinery built to preserve it.
+**Resolution**: FLUSH now invalidates only when it actually drained buffered writes (`drain.is_some()`); a no-write FLUSH is not a mutation (MutationAudit gets an explicit `discard_no_mutation`). Kill switch `AGENTFS_FUSE_FLUSH_INVAL=1`. After: 1,280/1,280 opens keep-cache, READs 1,280→64 (one cold read per file), stale rejections 0. 8/8 A/B pairs win, paired wall median 0.744.
+
+## 2026-06-11T12:35-07:00 — FOPEN_CACHE_DIR requires giving back the OPENDIR round trip
+**Type**: decision
+**Context**: readdirplus dominated handler time (482 calls × 30.6µs) because the kernel re-fetched directory contents on every scandir. Granting `FOPEN_CACHE_DIR|FOPEN_KEEP_CACHE` lets warm getdents hit the page cache, but the mount advertised `FUSE_NO_OPENDIR_SUPPORT`, so the kernel never sent OPENDIR and there was no reply to carry the flag.
+**Resolution**: `FUSE_NO_OPENDIR_SUPPORT` is now advertised only when dir caching is off (`AGENTFS_FUSE_CACHE_DIR=0`). Trade: one OPENDIR+RELEASEDIR round trip per opendir(3) (handler ~1.5µs) buys cached getdents for every warm re-listing — readdirplus 482→24 on the read profile, 2,858→1,425 on the git workload. Coherency: mount-local mutations notify the parent inode (kernel drops dir pages); cross-mount divergence is TTL-bounded like attrs.
+
+## 2026-06-11T12:40-07:00 — Read-path verdict: 4.0x, floor is the OPEN+FLUSH round-trip pair; next levers logged
+**Type**: deviation
+**Context**: Target was ≤1.5x. With READs and readdirplus mostly eliminated, each warm open/read/close cycle still pays two synchronous FUSE round trips (OPEN ~11µs handler + FLUSH ~1.6µs handler, ~60µs wall vs native ~14µs). FOPEN_NOFLUSH is ignored by the kernel under writeback cache (re-confirmed reasoning from the earlier spike), and connection-wide ENOSYS-on-FLUSH was evaluated and rejected for now: the per-fh write buffer tail would only land at async RELEASE, opening a stat-after-close staleness window.
+**Resolution**: 4.0x recorded honestly (3.2x better than the 12.7x start). Logged next levers, in order of expected value: (1) extend `keep_cache_for_read_open` beyond `Layer::Base` to upper/DB-backed files — requires relaxing the drift guard's sticky drop to fingerprint-based revalidation, since files created through the mount (git clone) currently lose eligibility permanently at first write; (2) FUSE passthrough for read fds (infrastructure counters already exist); (3) ENOSYS-on-FLUSH revisited only with a getattr-side pending-flush guarantee.
+
 ## 2026-06-11T10:25-07:00 — WS1 TTL hypothesis falsified by counters; warm-read target moves to WS2
 **Type**: surprise
 **Context**: The spec predicted raising entry/attr TTLs 1s→10s would fix the read-path warm steady-state (12.7x). Counter measurement shows request counts are IDENTICAL across TTL settings in the read benchmark (getattr 235, open 256, readdirplus 482, cold AND warm): the kernel already caches within iteration loops at 1s, and "warm" remounts, so every object pays exactly one round trip per mount regardless of TTL. Steady-state cost is ~1,229 requests x ~98us = per-request cost, not TTL expiry.