[datakit] Verify materialized fuzzy clusters

[ravwojdyla-agent]

• require a complete cluster-text artifact before verification starts
• solve grouped clusters in bounded distributed map tasks and route sparse markers back to normalized shards
• write the existing verified-marker Parquet schema with stable representative source keys
• keep truncated documents and exclude them from representative selection
• distinguish pipeline and cluster marker producers while preserving the common store input contract
• stacked on [datakit] Materialize fuzzy-duplicate cluster text #8724

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 923f19334a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve representatives across mid-cluster flushes

When a materialized cluster exceeds MAXIMUM_CLUSTER_CHARS (256 MiB), this branch solves the longest-sorted prefix and then discards all of its surviving representatives before processing the remainder. Consequently, shorter documents in later chunks are never compared with the cluster's longest representatives, producing false negatives and potentially retaining multiple copies that find_duplicates would remove if given the whole cluster. The memory guard must preserve earlier representatives across flushes or explicitly repartition the cluster before verification.

Useful? React with 👍 / 👎.