feat: Add VectorDBBench Cloud Leaderboard benchmark cases and client support

[jamesgao-jpg]

Summary

This PR adds the VectorDBBench Cloud Leaderboard benchmark surface. The goal is to complement the existing raw-performance leaderboard with cloud-oriented cases that capture production behaviors managed vector database users care about: ingest readiness, payload-aware search, tenant-shaped traffic, cold latency, and cost-aware interpretation.

What is added

New Cloud Leaderboard cases

• CloudPayloadSearchCase: measures search performance with explicit response payload profiles: ids_only, scalar_label, and vector. It supports unfiltered search, integer-filter search, and scalar-label filter search.
• CloudInsertCase: measures insert throughput and separates client insert completion from downstream readiness signals such as fully searchable and fully indexed.
• CloudColdLatencyCase: measures cold and warm serial latency so first-query and cache-sensitive serving behavior are visible instead of hidden by warm concurrent loops.
• CloudMultiTenantSearchCase: models SaaS-style tenant-routed workloads with deterministic tenant assignment and tenant-aware query routing.

Runtime and metric plumbing

• Threads payload profile configuration through case config, runners, client search calls, metrics, and JSON result output.
• Adds payload byte estimation to result metrics so payload-heavy searches can be compared more explicitly.
• Adds a cold/warm search runner and task-runner integration for cold latency measurement.
• Adds concurrent insert readiness polling for cloud insert runs.
• Adds first-class result fields for cloud insert, cold latency, payload profile, and related cloud case metadata.

Client support

• Milvus/Zilliz Cloud: adds payload output handling, scalar-label support, multitenant partition-key validation, and related schema checks.
• Pinecone: adds payload-profile search behavior, metadata/vector return handling, insert readiness polling through write/index LSNs where available, and namespace-based multitenant routing.
• turbopuffer: adds payload-profile search behavior, scalar payload label configuration, multitenant namespace support, write backpressure control, namespace pin/unpin CLI support for pinned benchmark runs, and explicit non-retryable partial tenant-write failure reporting.

CLI, frontend, and docs

• Adds CLI options for payload profile, cloud filters, cold query count, insert batch size/duration, tenant settings, and turbopuffer pinning.
• Adds frontend case config entries for the cloud payload search cases.
• Adds a May 2026 VectorDBBench Cloud Leaderboard release note under docs/release/2026-05-cloud-leaderboard.md.
• Updates README.md to mention the new Cloud Leaderboard benchmark cases and link to the release note.

Notes

• The raw Cloud Leaderboard result dump was intentionally removed from this PR. Result artifacts can be added later in a smaller, dedicated update or kept outside the source tree.
• Cloud cost and Pareto interpretation are described in the release note, but this PR focuses on benchmark case support and documentation rather than publishing a full result dataset.

Test Plan

• .venv/bin/python -m pytest tests/test_cloud_payload_search.py tests/test_cloud_payload_case.py tests/test_cloud_insert_case.py tests/test_cloud_cold_latency_case.py tests/test_multitenant_case.py tests/test_pinecone_multitenant.py tests/test_milvus.py -q -k 'not test_performance_1536d_50k'
• .venv/bin/python -m ruff check vectordb_bench/backend/clients/api.py vectordb_bench/backend/clients/turbopuffer/turbopuffer.py vectordb_bench/backend/runner/concurrent_runner.py vectordb_bench/cli/cli.py

[XuanYang-cn]

Requesting changes. I left the blocking findings inline.

[XuanYang-cn]

I found one additional required issue in the result read/write path.

These existing unresolved review threads still apply on the current head and should remain blockers: scalar-label payload loads with separated scalar_labels.parquet, Pinecone multitenant partial-insert accounting, cloud insert readiness polling without a timeout, and TurboPuffer --dry-run --pin-namespace mutating remote state before dry-run handling.

[XuanYang-cn]

I found two blocking issues that can still produce invalid Cloud Leaderboard measurements: scalar-label cases can reuse a collection loaded without scalar labels, and TurboPuffer multitenant optimize warms the base namespace instead of the tenant namespaces.

[XuanYang-cn]

Reviewed head a949875. I found blocking issues in the cloud benchmark state handling: multitenant case reuse can run against the wrong tenant layout, non-retryable partial insert failures do not stop the remaining insert workers, and Pinecone multitenant readiness counts are racy under concurrent insert.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jamesgao-jpg, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuanYang-cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment