Evaluate a stronger (local-first) embedding backend + pluggable embedder interface

[DanNegrut]

Context

PR #1 (@ashvin-a) proposed an opt-in OpenAI embedding backend plus a MongoDB->v2 index converter. We're not merging it for the public release:

1. Embeddings are on the retrieval hot path, so a cloud embedder needs a key + network on every query and breaks v2's local / keyless / offline design.
2. DENSE_FLOOR and the RRF weights are calibrated for bge-small's score scale; the PR swaps the embedder without recalibrating, so an OpenAI-embedded index would misfire the abstention/refuse logic.
3. The Mongo corpus is superseded by v2's fresher, structurally-chunked local index.

The reference implementation (the OpenAIEmbedder dispatch) is preserved in PR #1 for whoever picks this up.

The real open question

Is bge-small-en-v1.5 (384-dim) strong enough, or is retrieval recall the ceiling on answer quality? With a frontier generation model, retrieval is the bottleneck, so this is worth measuring rather than assuming.

Task (local-first, data-driven; defer until after the public release)

1. Add a small pluggable embedder interface (PR Integrate OpenAI embeddings created into V2 branch #1's dispatch is a fine starting point).
2. Build an index with a stronger local model (e.g. BAAI/bge-large-en-v1.5 ~1024-dim, or intfloat/e5-large-v2) via fastembed.
3. Benchmark recall@k / MRR / abstention on src/eval/gold.jsonl vs bge-small.
4. Recalibrate DENSE_FLOOR (and revisit RRF_K, PYCHRONO_BOOST) for the new score scale.
5. Report the quality delta and the cost (index size, query latency); decide whether to switch the default local embedder.

Stretch (only if the local benchmark shows embeddings genuinely cap quality): add a cloud (OpenAI) embedder behind the same interface as an explicit opt-in, with its own calibration.

cc @ashvin-a