[datakit] Materialize fuzzy-duplicate cluster text - #8724
[datakit] Materialize fuzzy-duplicate cluster text#8724ravwojdyla-agent wants to merge 1 commit into
Conversation
ravwojdyla-agent
commented
Aug 27, 2026
ravwojdyla-agent
commented
- join fuzzy-candidate attributes to normalized text once and group rows by cluster
- record source layout, split policy, affected member count, and completion in a versioned manifest
- identify oversized clusters with deterministic hash sampling; reject planner thresholds that leave clusters above the materializer cap unplanned
- use a MinHash partition for oversized components and record its Jaccard-based containment recall limit
- reject mismatched or repeated IDs; flag oversized text so verification keeps it
- stacked on [datakit] Solve complete fuzzy-duplicate clusters #8723
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dc992a8e47
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| splits = oversized.get(cluster_id, 1) | ||
| cluster_key = cluster_id | ||
| if splits > 1: | ||
| split_index = _split_hash(text) % splits |
There was a problem hiding this comment.
When an oversized component contains multiple documents above MAXIMUM_VERIFICATION_TEXT_CHARS, each document's text has already been replaced with "" before this call, so every truncated member receives the same split hash and lands in one partition regardless of splits. A mega-document-heavy component can therefore remain large enough to overload a reducer despite the configured cluster-size cap; derive a bounded hash from the raw text, or another content-bearing deterministic value, before discarding the stored text.
Useful? React with 👍 / 👎.
| fs, root = url_to_fs(counts_dir) | ||
| names = sorted(str(path).rsplit("/", 1)[-1] for path in fs.ls(root, detail=False) if str(path).endswith(".parquet")) |
There was a problem hiding this comment.
When this planner is rerun into an existing --out with fewer tasks, this directory listing also picks up old part-*.parquet files whose indices are no longer produced. Those stale counts are included in merged, inflating cluster sizes and split counts and potentially fragmenting verification differently from the current input; aggregate the paths returned in outcome.results or clear the counts directory before dispatch.
Useful? React with 👍 / 👎.
| summary = json.loads(summary_path.read_bytes()) | ||
| minimum_size = int(summary["minimum_size"]) | ||
| if minimum_size > max_cluster_size: |
There was a problem hiding this comment.
When --large-clusters accidentally points to a plan generated for another candidate artifact, this loader validates only minimum_size even though the planner records its candidates value in summary.json. The current candidates are then materialized using unrelated cluster IDs and sizes, leaving their actual oversized components unsplit and potentially exceeding the reducer cap; validate the plan's candidate identity against the resolved current candidate path before execution.
Useful? React with 👍 / 👎.
Join candidate attributes to normalized text once and store rows by cluster. Record source and shard provenance in a versioned manifest. Reject candidate layouts that do not match their normalized sources.
2ea5301 to
6ed5d5a
Compare
dc992a8 to
cabd5d8
Compare
2 participants
