[TST](garbage_collector): add MCMR hard delete test

[rescrv]

Description of changes

Verify the garbage collector hard-deletes a collection that spans two
regions. The test writes records in alternating rounds across both
clients, waits for compaction, deletes the collection, then tails the
GC pod logs via kubectl to confirm a "Hard deleting collections" entry
appears for the deleted collection UUID within a timeout.

Test plan

Is a test. CI.

Migration plan

N/A

Observability plan

N/A

Documentation Changes

N/A

Co-authored-by: AI

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Add MCMR GC hard-delete integration test with MinIO and GC log verification

This PR primarily adds a new multi-region property/integration-style test in chromadb/test/property/test_add_gc.py that validates garbage collection hard-delete behavior for a collection spanning two regions. The test alternates writes across two MCMR clients, waits for collection version increases (as a proxy for propagation/compaction progress), deletes the collection, tails garbage-collector pod logs via kubectl, and asserts a matching "Hard deleting collections" log entry for the target collection UUID.

It also updates CI/infra wiring to run this test in the cluster workflow and avoid accidental execution in incompatible jobs. Supporting changes include adding MinIO console port-forwarding in .github/actions/tilt/action.yaml, adding test_add_gc.py to cluster test matrices in .github/workflows/_python-tests.yml, excluding it from non-cluster property shards, and introducing a skip_single_region marker handling path in chromadb/test/conftest.py so tests can explicitly suppress classic single-region parametrization when multi-region mode is enabled.

This summary was automatically generated by @propel-code-bot

[propel-code-bot]

No issues were found; the PR appears robust and well-structured for validating multi-region hard delete behavior.

Status: No Issues Found | Risk: Low

Review Details

📁 33 files reviewed | 💬 0 comments

[tanujnay112]

can keep a vector/list of indexes and index into them to keep it compact

[rescrv]

done

[tanujnay112]

same here

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_add_gc_hard_deletes_mcmr_collection	View Logs