`loadParsedFilesForPaths` deserializes the whole ParsedFile store on every language pass before filtering by path — a 3-language repo JSON.parses the same ~480 MiB three times

[BecauseY]

Area

gitnexus (CLI / core / indexing / MCP server)

Summary

loadParsedFilesForPaths(storagePath, wantPaths) (gitnexus/src/storage/parsedfile-store.ts) reads every *.json shard in parsedfile-store/, JSON.parses it through the interning reviver, and only then keeps the entries whose filePath is in wantPaths. Scope-resolution calls it once per language (loadStoreFor(paths) in gitnexus/src/core/ingestion/scope-resolution/pipeline/phase.ts), so a repo with N languages deserializes the entire store N times — even when a language owns two files. Since 1.6.10 the parse phase calls it again for data-route-table files (parse-impl.ts), which adds one more full pass whenever such files exist.

Context

Same deployment as #3086: GitNexus as the indexing backend of a code-review service, Java-dominant monorepos, one file per worker job (GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=1). Measured on 1.6.10-rc.205; parsedfile-store.ts and phase.ts are unchanged in 1.6.10.

Expected behavior

A language pass should open only the shards that contain at least one wanted path. The cost of loading ParsedFiles for a language should be proportional to that language's files, not to the store size.

Actual behavior

On a ~6.3k-file monorepo (Java plus two languages with a single file each):

• 5,751 shards, ~480 MiB of JSON.
• The two single-file languages each deserialize all 5,751 shards to keep one entry; Java does it a third time.
• Before we changed anything else, the Java resolver's own work was ~28 s per pass while shard deserialization took over ten minutes per analyze — the outer JSON loop, not resolution, dominated "Linking symbols".
• With a per-shard path index in place (and our other local cache changes applied), removing only the index costs +8.3 s (63.3 s → 71.6 s); the remaining load is bounded by the shards that actually contain wanted paths. Output node/edge hashes identical.

Steps to reproduce

1. A Java repo with ~5k+ files that also contains a couple of files in other languages (e.g. one Python and one shell script).
2. GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES=1 gitnexus analyze --force <repo>, twice (second run warm).
3. Instrument loadParsedFilesForPaths (count readFile calls per invocation): every language pass reads shards.length files regardless of wantPaths.size.
4. Compare wall time of the minor-language passes with the Java pass: they are dominated by the same full-store deserialization.

Environment

• Linux x86_64 (glibc 2.38), Node v22.22.0
• gitnexus 1.6.10-rc.205 (loader and per-language call site unchanged in 1.6.10)

Logs / screenshots

Proposed fix (what we run locally)

Persist a per-shard path list next to each shard (<shard>.json.paths, one filePath per line) from both shard writers (persistParsedFileChunk / persistParsedFileChunkSync), and carry it along in restoreDurableParsedFileShard. The loader reads the sidecars first, opens only shards with at least one wanted path, and falls back to reading a shard whenever its sidecar is missing or corrupt — so stores written by older builds behave exactly as today and can be backfilled lazily. Shard JSON is untouched.

An equivalent alternative is a store-level index.json (path → shardId) written when the parse phase completes; the per-shard sidecar is simpler to keep consistent with the durable content-addressed cache.

Logs

Can attach a readFile-count trace per language pass and the timing breakdown on request.