(diversity) Integrate determinant diversity in tokp full precision search and disk search by narendatha · Pull Request #1011 · microsoft/DiskANN

narendatha · 2026-05-04T12:30:38Z

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Wires the determinant-diversity post-processing algorithm as an optional SearchPlugin for two search paths:

Async graph-index (full-precision topk via inmem::FullAccessor)
Disk-index (via DiskIndexSearcher post-processor)

Determinant-diversity reranks search results to increase geometric diversity among the top-k, at a configurable quality/speed tradeoff.

Motivation

Standard nearest neighbor search often returns highly similar/redundant results. For some scenarios like RAG search, it's more valuable to retrieve diverse, complementary documents that cover different aspects of the query. This PR implements a principled approach based on determinant maximization of the Gram matrix of query-scaled vectors.

Algorithm

The algorithm scales each candidate vector by (similarity_to_query)^power and then greedily selects vectors to maximize the determinant of the Gram matrix (equivalent to maximizing the volume of the parallelepiped spanned by the vectors).

Two variants are implemented:

eta = 0: Pure greedy orthogonalization (pivoted QR decomposition)
eta > 0: Ridge-regularized version that balances diversity with numerical stability

Both variants run in O(n k d) time where n = candidates, k = results to return, d = dimension.

Performance Characteristics

Tested on OpenAI embeddings dataset (3072 dimensions, 10K sample):

Metric	Normal Search	Det_Div Search	Notes
QPS	~150	~130	~13% overhead for diversity optimization
Recall@10	98-99%	87-88%	Expected trade-off for diversity

The recall reduction is expected - Det_Div search intentionally selects diverse vectors rather than the absolute nearest neighbors.

Diversity Improvement

We measure diversity using log-determinant of the Gram matrix of scaled vectors (higher = vectors span more volume = more diverse).

RAG search consistently improves diversity (all improvements >= 0)
Average improvement of +2.9 to +3.3 in log_det corresponds to 10-27x more volume spanned by selected vectors

Parameter Tuning Guide

`power` (default: 2.0)

Controls the trade-off between relevance and diversity:

Higher values (e.g., 3.0+): Strongly prefer relevance - vectors closer to the query get much higher weight, resulting in less diversity but higher recall
Lower values (e.g., 1.0): Prefer diversity over relevance - similarity weighting is more uniform, allowing more diverse vectors to be selected
Value of 0: Pure diversity maximization with no relevance weighting

`eta` (default: 0.01)

Ridge regularization parameter that controls numerical robustness and relevance preference:

Higher values (e.g., 0.1+): More robust numerically, but biases toward selecting high-relevance vectors over diverse ones
Lower values (e.g., 0.001): Closer to pure determinant maximization, maximizes diversity but may be less stable
Value of 0: Falls back to pure greedy orthogonalization (no ridge regularization)

Recommended Settings

Use Case	`power`	`eta`
Balanced (default)	2.0	0.01
High diversity	1.0	0.001
High relevance	3.0	0.1

Changes

Core algorithm (`diskann-providers`)

Added determinant_diversity_post_process() in diskann-providers/src/model/graph/provider/async_/determinant_diversity_post_process.rs
Two selection strategies controlled by eta:
- eta == 0: greedy orthogonalization
- eta > 0: eta-weighted residual selection

Async graph-index path (`diskann-benchmark`)

Added FullPrecisionVectorAccessor shim trait + impl for inmem::FullAccessor<f32, ...> so the post-processor can fetch real candidate vectors
Implemented DeterminantDiversity plugin that satisfies glue::SearchPostProcess using for<'a, 'b> SearchStrategy HRTB bound (credit: @hildebrandmw)
Wired run_topk_timed helper to expose QPS/latency/recall metrics in the benchmark output

Disk-index path (`diskann-disk`, `diskann-benchmark`)

Added DeterminantDiversityAndFilter processor to DiskSearchPostProcessor enum
Extended searcher.search() to accept an optional SearchPostProcessorKind::DeterminantDiversity { power, eta }
Extended DiskSearchPhase JSON input to accept post_processor
Result-counting uses result_count from search stats to correctly bound post-processed results

Input/config

New TopkPostProcessor serde enum in diskann-benchmark/src/inputs/post_processor.rs with validation (power > 0, eta >= 0)
Two new example configs: async-determinant-diversity.json, disk-index-determinant-diversity.json

Benchmark JSON examples

// async graph-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.0 }

// disk-index
"post_processor": { "type": "determinant-diversity", "power": 1.0, "eta": 0.5 }

Notes

Determinant-diversity is only wired for full-precision topk on the async path; quantized (PQ/scalar) paths are intentionally excluded
Spherical, range, and filtered-search plugin variants are unchanged; this post-processor is topk-only

…ugins

Replace kind()-based string equality checks with explicit is_match() and get() phase-shape helpers on plugin structs. This avoids fragile ordering assumptions and makes each plugin responsible for recognising its own phase shape.

Co-authored-by: Copilot <copilot@github.com>

…plugins # Conflicts: # diskann-benchmark/src/backend/index/benchmarks.rs # diskann-benchmark/src/backend/index/product.rs # diskann-benchmark/src/backend/index/scalar.rs # diskann-benchmark/src/backend/index/search/plugins.rs # diskann-benchmark/src/backend/index/spherical.rs # diskann-benchmark/src/inputs/graph_index.rs # diskann-benchmark/src/inputs/mod.rs

Co-authored-by: Copilot <copilot@github.com>

…ojection issue This commit explores approaches to wire real candidate vectors into async determinant-diversity post-processing. Current state: IN COMPILATION ERROR (intentional for analysis) Attempted approaches: 1. Initial shim-trait FullPrecisionVectorAccessor with async get_full_precision_vector() - Resulted in 'implementation not general enough' at search_with() call 2. Removed explicit for<'a> post_processor::DeterminantDiversity bound - Still fails - the constraint is inherent in search_with() signature itself Root cause analysis: - search_with() requires: PP: for<'a> SearchPostProcess<S::SearchAccessor<'a>, T, O> - This means post-processor must work for ANY accessor lifetime 'a - But query = queries.row(query_idx) is borrowed for specific loop iteration lifetime - These are fundamentally incompatible - a borrowed value can't satisfy for<'a> generically Compiler errors (3 total): - 'not general enough': implementation needed for or<'a> but found specific '0 - 'does not live long enough': queries lifetime too short for 'static requirement Files modified: - diskann-benchmark/src/backend/index/benchmarks.rs: * Removed explicit for<'a> post_processor::DeterminantDiversity constraint * Narrowed plugin impl to FullPrecisionProvider<f32> - diskann-benchmark/src/backend/index/post_processor/determinant_diversity.rs: * Added shim trait FullPrecisionVectorAccessor * Async method get_full_precision_vector(&mut self, id) -> impl Future<...> Next steps to investigate: - Move determinant-diversity outside search_with() as post-processing reranking - This avoids HRTB entirely by applying after candidates are returned - Benchmark impact: measure recall/QPS with external reranking vs baseline Related context: - Disk index determinant-diversity works correctly (uses real vectors, shows 51-53% QPS cost) - Shared algorithm fixed (distance-to-similarity scoring direction) - Branch already merged with origin/main

Co-authored-by: Copilot <copilot@github.com>

…educe duplication - Use for<'a, 'b> SearchStrategy bound (user-provided fix) to break HRTB lifetime projection issue in the search_with post-processor constraint - Wire FullPrecisionVectorAccessor shim trait so async det-div post-processor fetches real candidate vectors instead of placeholder distances - Populate QPS/latency metrics in async det-div benchmark path (previously all 'missing') - Extract run_topk_timed helper to eliminate ~100 lines of duplicated loop/timing/recall machinery from DeterminantDiversity::run - Update async-determinant-diversity.json example tag (async-index-build -> graph-index-build) - Fix clippy::manual_async_fn in FullPrecisionVectorAccessor shim trait

codecov-commenter · 2026-05-06T05:22:20Z

Codecov Report

❌ Patch coverage is 52.48447% with 306 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.21%. Comparing base (a47a1e6) to head (85797ce).

Files with missing lines	Patch %	Lines
diskann-disk/src/search/provider/disk_provider.rs	18.18%	90 Missing ⚠️
diskann-benchmark/src/backend/index/benchmarks.rs	8.75%	73 Missing ⚠️
...kend/index/post_processor/determinant_diversity.rs	0.00%	36 Missing ⚠️
...kann-benchmark/src/backend/index/search/plugins.rs	41.30%	27 Missing ⚠️
diskann-benchmark-core/src/search/graph/knn.rs	4.16%	23 Missing ⚠️
diskann-benchmark/src/inputs/graph_index.rs	55.26%	17 Missing ⚠️
diskann-providers/src/post_processor.rs	70.45%	13 Missing ⚠️
diskann-benchmark/src/inputs/disk.rs	0.00%	11 Missing ⚠️
.../src/model/graph/provider/determinant_diversity.rs	96.73%	8 Missing ⚠️
diskann-benchmark/src/inputs/post_processor.rs	0.00%	7 Missing ⚠️
... and 1 more

❌ Your patch status has failed because the patch coverage (52.48%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1011      +/-   ##
==========================================
- Coverage   89.46%   89.21%   -0.25%     
==========================================
  Files         459      463       +4     
  Lines       85482    86087     +605     
==========================================
+ Hits        76474    76805     +331     
- Misses       9008     9282     +274

Flag	Coverage Δ
miri	`89.21% <52.48%> (-0.25%)`	⬇️
unittests	`88.84% <52.48%> (-0.25%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark/src/backend/index/mod.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/backend/index/search/knn.rs	`98.00% <100.00%> (+21.43%)`	⬆️
diskann-benchmark/src/backend/index/spherical.rs	`100.00% <ø> (ø)`
diskann-benchmark/src/inputs/mod.rs	`64.28% <ø> (ø)`
diskann-disk/src/build/builder/core.rs	`95.24% <100.00%> (+<0.01%)`	⬆️
diskann-tools/src/utils/search_disk_index.rs	`0.00% <0.00%> (ø)`
diskann-benchmark/src/inputs/post_processor.rs	`0.00% <0.00%> (ø)`
.../src/model/graph/provider/determinant_diversity.rs	`96.73% <96.73%> (ø)`
diskann-benchmark/src/inputs/disk.rs	`1.45% <0.00%> (-0.09%)`	⬇️
diskann-providers/src/post_processor.rs	`70.45% <70.45%> (ø)`
... and 6 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Use for<'a, 'b> SearchStrategy bound to resolve HRTB lifetime mismatch - Wire FullPrecisionVectorAccessor shim trait for async det-div to fetch real vectors - Implement DeterminantDiversity post-processor for async graph-index path - Extract run_topk_timed helper to eliminate ~100 lines of code duplication - Wire post_processor parameter to disk-index search pipeline - Update search parameter handling and result counting for post-processed results - Add TopkPostProcessor input type and necessary imports - Populate QPS/latency metrics in async det-div benchmark path

…rosoft/DiskANN into u/narendatha/det_div_plugins

Co-authored-by: Copilot <copilot@github.com>

hildebrandmw

Thanks Naren! My biggest concern is diskann-benchmark. I would really prefer if we generalized diskann-benchmark-core to handle post-processing, which would reduce bugs and be more broadly applicable.

On the implementation front, there's a bit of work to do regarding being more opinionated on the nature of inputs, establishing invariants of the diversity search with a bespoke struct (which could help with the previous point), reducing code duplication, testing the implementation, and removing allocations with a Matrix.

I'll leave reviewing the diskann-disk changes to others.

…tr() methods

- Add 4th type parameter PP to KNN with default of () - Add with_postprocessor() constructor for post-processor support - Keep new() for standard KNN without post-processing - Add accessor methods for index, queries, strategy, post_processor - Update Search impl to work with generic KNN<DP, T, S, ()> - Maintains backward compatibility - all existing code continues to work - Provides foundation for DeterminantDiversity plugin refactoring

…diskann-providers - Add diskann-providers/src/post_processor.rs with centralized DeterminantDiversityParams - Define DeterminantDiversityError for precise error reporting - Provide unified validation: power > 0.0, eta >= 0.0 - Add accessor methods and Display impl - Import tests included - Update diskann-benchmark to use shared type from diskann-providers - Remove duplicate local definition in plugins.rs - Maintains backward compatibility with anyhow::Error conversion - Foundation for sharing post-processor logic across subsystems (benchmark, disk-index)

…process.rs - Document algorithm overview, parameters (power, eta), variants, time complexity - Clarify distinction between eta=0 (exact) and eta>0 (ridge-regularized) paths - Reference asymptotic complexity O(m^3) due to determinant computation

- test_diversity_selects_orthogonal_candidates: Verify orthogonal pair chosen over parallel - test_diversity_selects_orthogonal_candidates_with_eta: Same test for eta>0 variant - test_high_power_prefers_closer_candidates: Verify relevance weighting with high power - test_equal_distances: Verify stable behavior with equal-distance candidates - test_eta_zero_is_greedy_path: Confirm eta=0.0 routes to greedy orthogonalization Note: Diversity tests use equal distances to eliminate relevance weighting interference, making them pure tests of the geometric diversity property.

- Remove duplicate post_process_with_eta_f32 and post_process_greedy_orthogonalization_f32 - Unify into single greedy_orthogonal_select() with inv_sqrt_eta parameter - inv_sqrt_eta=1.0/sqrt(eta) for ridge-regularized path (eta>0) - inv_sqrt_eta=1.0 for exact greedy path (eta=0) - Eliminates ~80 lines of near-duplicate code - All 14 existing tests continue to pass

- Use contiguous Matrix allocation instead of separate Vec per candidate - Reduces heap allocations from O(n) to O(1) where n = num candidates - Improves cache locality during orthogonalization iteration - Access residuals via row(i) / row_mut(i) instead of indexing Vec of Vecs - All 14 tests continue to pass

- Move implementation to model/graph/provider/determinant_diversity.rs - Export via provider::mod.rs as determinant_diversity_post_process - Keep backward-compatible re-export from async_::mod.rs - Remove old async_-scoped source file - Verify build and full diskann-providers tests pass

…plugins

Mark Hildebrand and others added 12 commits April 28, 2026 14:47

Make benchmarks stateful.

dcdc377

Checkpoint.

d60ac07

Checkopint.

6cc92bd

Oh boy.

f050e3d

Merge remote-tracking branch 'origin/main' into mhildebr/benchmark-pl…

2d03d11

…ugins

Fix delimiters.

aa78836

Revert formatting.

ccf891a

Make plugins slightly more flexible.

2e060ea

Merge branch 'main' into mhildebr/benchmark-plugins

7a85df8

Add determinant-diversity plugin support on search-plugin architecture

fc8008a

Integrate determinant-diversity via disk search_with post-processor

298d1b8

Restrict determinant-diversity to async full-precision topk

fbd34fd

narendatha changed the base branch from main to mhildebr/benchmark-plugins May 4, 2026 12:41

Keep single async determinant-diversity example JSON

851174e

Base automatically changed from mhildebr/benchmark-plugins to main May 5, 2026 00:45

narendatha and others added 10 commits May 5, 2026 12:41

Use SearchPhaseKind::as_str in benchmark plugin kinds

3a6aa1a

remove serde defaults

ccfe4d7

Co-authored-by: Copilot <copilot@github.com>

minor merge fix

86883b7

hook up actual algorithm, replace placeholder.

554bc7f

Co-authored-by: Copilot <copilot@github.com>

apply mark's beautiful fix for lifetime issue

b73abc8

Co-authored-by: Copilot <copilot@github.com>

Fix CI clippy-features spherical plugin errors and apply formatting

701ce8e

narendatha and others added 4 commits May 6, 2026 16:57

Merge branch 'u/narendatha/det_div_plugins' of https://github.com/mic…

6f47ba3

…rosoft/DiskANN into u/narendatha/det_div_plugins

imrpove code coverage

8acbba2

minor fix

a48e255

Co-authored-by: Copilot <copilot@github.com>

narendatha changed the title ~~(draft) integrate determinant diversity in tokp full precision search and disk search~~ (diversity) Integrate determinant diversity in tokp full precision search and disk search May 6, 2026

narendatha marked this pull request as ready for review May 6, 2026 15:03

narendatha requested review from a team and Copilot May 6, 2026 15:03

Copilot started reviewing on behalf of narendatha May 6, 2026 15:04 View session

cargo fmt

468b5d2

hildebrandmw reviewed May 8, 2026

View reviewed changes

narendatha added this to DiskANN backlog May 12, 2026

narendatha moved this to In Progress in DiskANN backlog May 12, 2026

narendatha removed this from DiskANN backlog May 12, 2026

narendatha linked an issue May 12, 2026 that may be closed by this pull request

Implement determinant diversity search #909

Open

narendatha added 15 commits May 13, 2026 12:58

WIP: Benchmarks refactoring - threading fix, rich struct params, as_s…

d9e66ba

…tr() methods

Refactor determinant-diversity benchmark path

ca20a24

cargo fmt and clippy fixes for CI

f538583

Use shared determinant-diversity params validation

f58789d

Merge remote-tracking branch 'origin/main' into u/narendatha/det_div_…

79c635f

…plugins

code review comment, use a struct instead of a tuple

3c11d36

Refine determinant-diversity invariants and range representation

b65e673

minor code cleanup

85797ce

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011

(diversity) Integrate determinant diversity in tokp full precision search and disk search#1011
narendatha wants to merge 43 commits into
mainfrom
u/narendatha/det_div_plugins

narendatha commented May 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 6, 2026 •

edited

Loading

Uh oh!

hildebrandmw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

narendatha commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integrate determinant-diversity into topk full-precision search and disk search

Summary

Motivation

Algorithm

Performance Characteristics

Diversity Improvement

Parameter Tuning Guide

power (default: 2.0)

eta (default: 0.01)

Recommended Settings

Changes

Core algorithm (diskann-providers)

Async graph-index path (diskann-benchmark)

Disk-index path (diskann-disk, diskann-benchmark)

Input/config

Benchmark JSON examples

Notes

Uh oh!

codecov-commenter commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hildebrandmw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

narendatha commented May 4, 2026 •

edited

Loading

`power` (default: 2.0)

`eta` (default: 0.01)

Core algorithm (`diskann-providers`)

Async graph-index path (`diskann-benchmark`)

Disk-index path (`diskann-disk`, `diskann-benchmark`)

codecov-commenter commented May 6, 2026 •

edited

Loading