Skip to content

Add Cargo/pyproject/tag version-consistency check - #63

Merged
moeyensj merged 1 commit into
mainfrom
check-versions
Apr 22, 2026
Merged

Add Cargo/pyproject/tag version-consistency check#63
moeyensj merged 1 commit into
mainfrom
check-versions

Conversation

@moeyensj

@moeyensj moeyensj commented Apr 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Preventing a recurrence of today's v2.0.0rc6 publish failure: crates.io got the right version but PyPI 400-rejected the upload because pyproject.toml was still pinned at "2.0.0rc5" (maturin built difi-2.0.0rc5-*.whl wheels against an rc6 Cargo.toml). Same class of drift explains why v2.0.0rc4 never got tagged — silent config mismatch, failed publish, move on.

This wires a single-source-of-truth check into both the PR gate and the publish pre-flight so Cargo ↔ pyproject ↔ tag drift can never reach a release again.

Changes

  • scripts/check_versions.sh — compares Cargo.toml (SemVer, 2.0.0-rc6) to pyproject.toml (PEP 440, 2.0.0rc6) on the PEP 440 normal form. Optional tag arg verifies tag-vs-both. Emits ::error:: annotations in CI and a plain ✓ line locally.
  • rust.yml — Build & Lint: runs the check right after checkout, no tag arg. Drift caught at PR time.
  • publish.yml — new preflight job: runs the check with ${GITHUB_REF_NAME} as the tag, gates publish-crate / build-wheels / build-sdist via needs: preflight. Inconsistent tree → no crates.io or PyPI upload.
  • pyproject.toml 2.0.0rc5 → 2.0.0rc6: pyproject catches up to main's Cargo.toml so this PR lands green. Does not re-publish (rc6 is live on crates.io; the PyPI slot for rc6 stays skipped since rc5 occupies the only path to it). The next tag — rc7, landing with PR Add multi-partition DIFI, CIFI-output reuse, and IgnoredLinkages #62 — is the first to exercise the preflight end-to-end.

Test plan

  • Local: ./scripts/check_versions.sh → ✓ (versions match after pyproject bump)
  • Local: ./scripts/check_versions.sh v2.0.0rc6 → ✓ (tag matches)
  • Local: ./scripts/check_versions.sh v2.0.0rc7 → exit 1, two ::error:: annotations (tag doesn't match either file)
  • CI: Build & Lint job surfaces the step with the ✓ line (expected green on this PR)
  • Reviewer: confirm the preflight needs: preflight is sufficient to block artifact upload — matrix job build-wheels inherits it for both OS variants
  • Reviewer: any interest in also running ./scripts/check_versions.sh in a pre-commit / pre-push hook? Low-priority follow-up

🤖 Generated with Claude Code

@moeyensj @claude
Add version-consistency check to guard Cargo/pyproject/tag drift
8a56a4d 
Today's rc6 publish partially failed: crates.io published difi-rs
v2.0.0-rc6 fine, but PyPI 400-rejected the upload because pyproject.toml
was still pinned at "2.0.0rc5" (from the prior release). maturin built
wheels named difi-2.0.0rc5-*.whl against an rc6 Cargo.toml, PyPI said
"File already exists", and the Publish workflow failed. The same class of
drift also explains why v2.0.0rc4 never got a tag (publish failed mid-way
and was abandoned).

This adds a shared check and wires it into both workflows so the drift
cannot reach a release again.

- `scripts/check_versions.sh` — single source of truth. Compares
  Cargo.toml (SemVer, e.g. `2.0.0-rc6`) against pyproject.toml (PEP 440,
  e.g. `2.0.0rc6`) on the PEP 440 normal form. Optionally takes a tag
  argument and verifies it matches both. Emits `::error::` annotations
  for GitHub Actions and a plain ✓ line for local use.

- `rust.yml` — runs the check on every PR inside the existing
  build-lint job, immediately after checkout. Drift is caught at PR time,
  not release time.

- `publish.yml` — new `preflight` job runs the check against the pushed
  tag before any artifact-producing job starts. publish-crate,
  build-wheels, and build-sdist all `needs: preflight`, so
  crates.io/PyPI cannot receive a build from an inconsistent tree.

Also bumps `pyproject.toml` from rc5 → rc6 so the check passes on this
PR and pyproject matches the rc6 state of Cargo.toml on main. This does
not re-publish rc6 (rc6 is live on crates.io and the PyPI slot is blocked
by rc5); the next release tag — rc7, when partitions lands — will be the
first to exercise the preflight end-to-end.

Verified locally:
- `./scripts/check_versions.sh` → pass
- `./scripts/check_versions.sh v2.0.0rc6` → pass
- `./scripts/check_versions.sh v2.0.0rc7` → fail (exit 1, two error lines)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@moeyensj
moeyensj merged commit d680bfa into main Apr 22, 2026
4 checks passed
@moeyensj
moeyensj deleted the check-versions branch April 22, 2026 19:32
moeyensj added a commit that referenced this pull request Apr 22, 2026
@moeyensj @claude
Add multi-partition DIFI, CIFI-output reuse, and IgnoredLinkages
c53440b 
The CLI's analyze-linkages was single-partition only because the library's
update_all_objects collapsed AllObjects rows by object_id, overwriting rows
when the same object appeared across partitions. This lifts that restriction,
adds CIFI-output reuse, and — because difi is designed to validate science
algorithms — makes the "phantom classification" failure mode explicit rather
than silent.

Library (breaking changes to public signatures):
- src/difi.rs: update_all_objects takes &PartitionSummary and filters
  obj_to_idx to rows matching the partition; multi-partition AllObjects is
  now safe to loop over.
- src/difi.rs: classify_linkages and analyze_linkages return
  (AllLinkages, IgnoredLinkages). Linkages with zero observations inside
  the target partition are excluded from AllLinkages and reported in
  IgnoredLinkages with reason NoObservationsInPartition, instead of being
  classified as phantom pure/contaminated/mixed rows with
  num_obs_outside_partition == num_obs. Callers loop and aggregate.
- src/types.rs: new IgnoredLinkage, IgnoredLinkageReason, IgnoredLinkages
  types, symmetric with AllLinkages.
- src/io.rs: new readers read_all_objects / read_partition_summaries /
  read_findable_observations (for reuse); new writer write_ignored_linkages.
  Readers threading interned strings take &mut StringInterner; doc comments
  spell out the observations-first ordering contract.
- src/python/mod.rs: destructures the new tuple; returns num_ignored_linkages
  in the result dict. Python parity tests stay bit-for-bit green
  (single-partition → ignored table is empty).

CLI:
- analyze-linkages exposes PartitionArgs and loops DIFI over each partition;
  all_linkages.parquet is flat with a partition_id column.
- --cifi-output-dir reuses a prior CIFI snapshot (skips the CIFI phase,
  loads all_objects / partition_summaries / findable_observations, runs
  DIFI per partition). Mutually exclusive with partition flags; SHA-256
  prefix fingerprint check against the reused manifest aborts on mismatch.
  Manifest gains a reused_cifi provenance field.
- When any linkages are ignored, the CLI writes ignored_linkages.parquet
  and emits a structured warning to stderr (and NDJSON under
  --progress-json). The manifest's new warnings section records
  ignored_linkage_rows and orphan_linkages counts — orphan linkages
  (never classified in any partition) are a strong signal of a
  mismatched --linkages file for the partition scheme.
- Conflict rule: --cifi-output-dir + any non-default partition flag
  is rejected up front.
- README documents multi-partition analyze-linkages, the reuse flow, the
  ignored_linkages semantics, the output layout, and the
  ./scripts/check_versions.sh developer invocation.

Tests (57 pass, +11 new over v1 CLI):
- Library units: multi-partition update_all_objects isolation;
  per-partition sequential accumulation; classify_linkages excludes
  wholly-external linkages and surfaces them in IgnoredLinkages.
- IO roundtrip: read_all_objects / read_partition_summaries /
  read_findable_observations round-trip (with interner re-alignment).
- CLI: sliding-partition analyze-linkages produces multi-partition
  output; --cifi-output-dir succeeds and records provenance;
  ignored_linkages.parquet + manifest warnings populated for sliding
  partitions; fingerprint mismatch and partition-flag conflicts
  rejected.
- Python parity tests remain byte-for-byte green.

Version: Cargo.toml 2.0.0-rc6 -> 2.0.0-rc7, pyproject.toml 2.0.0rc6 ->
2.0.0rc7 (kept in lockstep by the preflight check from PR #63).

Verified:
- cargo build --no-default-features
- cargo build --features cli
- cargo fmt -- --check
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --features cli  # 57 passed
- ./scripts/check_versions.sh  # versions consistent
- ./scripts/check_versions.sh v2.0.0rc7  # tag + versions consistent

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@moeyensj moeyensj mentioned this pull request Apr 22, 2026
Merged
11 tasks
moeyensj added a commit that referenced this pull request Apr 22, 2026
@moeyensj @claude
Add multi-partition DIFI, CIFI-output reuse, and IgnoredLinkages
383766f 
The CLI's analyze-linkages was single-partition only because the library's
update_all_objects collapsed AllObjects rows by object_id, overwriting rows
when the same object appeared across partitions. This lifts that restriction,
adds CIFI-output reuse, and — because difi is designed to validate science
algorithms — makes the "phantom classification" failure mode explicit rather
than silent.

Library (breaking changes to public signatures):
- src/difi.rs: update_all_objects takes &PartitionSummary and filters
  obj_to_idx to rows matching the partition; multi-partition AllObjects is
  now safe to loop over.
- src/difi.rs: classify_linkages and analyze_linkages return
  (AllLinkages, IgnoredLinkages). Linkages with zero observations inside
  the target partition are excluded from AllLinkages and reported in
  IgnoredLinkages with reason NoObservationsInPartition, instead of being
  classified as phantom pure/contaminated/mixed rows with
  num_obs_outside_partition == num_obs. Callers loop and aggregate.
- src/difi.rs: "found" semantics tightened to in-partition counts.
  found_pure = pure && obs_inside_partition >= min_obs (was total_obs),
  found_contaminated = contaminated && dominant_count_in_partition >= min_obs
  (was whole-linkage dominant_count). analyze_linkages's found_objects set
  uses found_pure rather than pure. Prevents cross-boundary linkages with
  enough total obs but few inside any one partition from inflating
  partition_summary.completeness above 100%. Single-partition runs are
  unaffected (obs_inside_partition == total_obs there); Python parity
  tests remain bit-for-bit green.
- src/types.rs: new IgnoredLinkage, IgnoredLinkageReason, IgnoredLinkages
  types, symmetric with AllLinkages.
- src/io.rs: new readers read_all_objects / read_partition_summaries /
  read_findable_observations (for reuse); new writer write_ignored_linkages.
  Readers threading interned strings take &mut StringInterner; doc comments
  spell out the observations-first ordering contract.
- src/python/mod.rs: destructures the new tuple; returns num_ignored_linkages
  in the result dict. Python parity tests stay bit-for-bit green
  (single-partition → ignored table is empty).

CLI:
- analyze-linkages exposes PartitionArgs and loops DIFI over each partition;
  all_linkages.parquet is flat with a partition_id column.
- --cifi-output-dir reuses a prior CIFI snapshot (skips the CIFI phase,
  loads all_objects / partition_summaries / findable_observations, runs
  DIFI per partition). Mutually exclusive with partition flags; SHA-256
  prefix fingerprint check against the reused manifest aborts on mismatch.
  Manifest gains a reused_cifi provenance field. In reuse mode the
  manifest's scenarios[0].metric and scenarios[0].partitions now reflect
  the reused snapshot's scheme rather than the (defaulted) CLI args, so
  reproducibility metadata is accurate.
- When any linkages are ignored, the CLI writes ignored_linkages.parquet
  and emits a structured warning to stderr (and NDJSON under
  --progress-json). The manifest's new warnings section records
  ignored_linkage_rows and orphan_linkages counts — orphan linkages
  (never classified in any partition) are a strong signal of a
  mismatched --linkages file for the partition scheme.
- Conflict rule: --cifi-output-dir + any non-default partition flag
  is rejected up front.
- README documents multi-partition analyze-linkages, the reuse flow, the
  ignored_linkages semantics, the output layout, the updated in-partition
  "found" definition of completeness, and the ./scripts/check_versions.sh
  developer invocation.

Tests (58 pass, +12 new over v1 CLI):
- Library units: multi-partition update_all_objects isolation;
  per-partition sequential accumulation; classify_linkages excludes
  wholly-external linkages and surfaces them in IgnoredLinkages.
- IO roundtrip: read_all_objects / read_partition_summaries /
  read_findable_observations round-trip (with interner re-alignment).
- CLI: sliding-partition analyze-linkages produces multi-partition
  output; --cifi-output-dir succeeds and records provenance;
  reuse manifest captures the snapshot's partition scheme (regression
  test against the reuse-defaults bug found during empirical validation);
  ignored_linkages.parquet + manifest warnings populated for sliding
  partitions; fingerprint mismatch and partition-flag conflicts
  rejected.
- Python parity tests remain byte-for-byte green.

Version: Cargo.toml 2.0.0-rc6 -> 2.0.0-rc7, pyproject.toml 2.0.0rc6 ->
2.0.0rc7 (kept in lockstep by the preflight check from PR #63).

Verified:
- cargo build --no-default-features
- cargo build --features cli
- cargo fmt -- --check
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --features cli  # 58 passed
- ./scripts/check_versions.sh  # versions consistent
- ./scripts/check_versions.sh v2.0.0rc7  # tag + versions consistent
- Empirical validation on 166M-row neomod survey with 12k synthetic linkages

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
moeyensj added a commit that referenced this pull request Apr 23, 2026
@moeyensj @claude
Add multi-partition DIFI, CIFI-output reuse, and IgnoredLinkages
276e045 
The CLI's analyze-linkages was single-partition only because the library's
update_all_objects collapsed AllObjects rows by object_id, overwriting rows
when the same object appeared across partitions. This lifts that restriction,
adds CIFI-output reuse, and — because difi is designed to validate science
algorithms — makes the "phantom classification" failure mode explicit rather
than silent.

Library (breaking changes to public signatures):
- src/difi.rs: update_all_objects takes &PartitionSummary and filters
  obj_to_idx to rows matching the partition; multi-partition AllObjects is
  now safe to loop over.
- src/difi.rs: classify_linkages and analyze_linkages return
  (AllLinkages, IgnoredLinkages). Linkages with zero observations inside
  the target partition are excluded from AllLinkages and reported in
  IgnoredLinkages with reason NoObservationsInPartition, instead of being
  classified as phantom pure/contaminated/mixed rows with
  num_obs_outside_partition == num_obs. Callers loop and aggregate.
- src/difi.rs: "found" semantics tightened to in-partition counts.
  found_pure = pure && obs_inside_partition >= min_obs (was total_obs),
  found_contaminated = contaminated && dominant_count_in_partition >= min_obs
  (was whole-linkage dominant_count). analyze_linkages's found_objects set
  uses found_pure rather than pure. Prevents cross-boundary linkages with
  enough total obs but few inside any one partition from inflating
  partition_summary.completeness above 100%. Single-partition runs are
  unaffected (obs_inside_partition == total_obs there); Python parity
  tests remain bit-for-bit green.
- src/types.rs: new IgnoredLinkage, IgnoredLinkageReason, IgnoredLinkages
  types, symmetric with AllLinkages.
- src/io.rs: new readers read_all_objects / read_partition_summaries /
  read_findable_observations (for reuse); new writer write_ignored_linkages.
  Readers threading interned strings take &mut StringInterner; doc comments
  spell out the observations-first ordering contract.
- src/python/mod.rs: destructures the new tuple; returns num_ignored_linkages
  in the result dict. Python parity tests stay bit-for-bit green
  (single-partition → ignored table is empty).

CLI:
- analyze-linkages exposes PartitionArgs and loops DIFI over each partition;
  all_linkages.parquet is flat with a partition_id column.
- --cifi-output-dir reuses a prior CIFI snapshot (skips the CIFI phase,
  loads all_objects / partition_summaries / findable_observations, runs
  DIFI per partition). Mutually exclusive with partition flags; SHA-256
  prefix fingerprint check against the reused manifest aborts on mismatch.
  Manifest gains a reused_cifi provenance field. In reuse mode the
  manifest's scenarios[0].metric and scenarios[0].partitions now reflect
  the reused snapshot's scheme rather than the (defaulted) CLI args, so
  reproducibility metadata is accurate.
- When any linkages are ignored, the CLI writes ignored_linkages.parquet
  and emits a structured warning to stderr (and NDJSON under
  --progress-json). The manifest's new warnings section records
  ignored_linkage_rows and orphan_linkages counts — orphan linkages
  (never classified in any partition) are a strong signal of a
  mismatched --linkages file for the partition scheme.
- Conflict rule: --cifi-output-dir + any non-default partition flag
  is rejected up front.
- README documents multi-partition analyze-linkages, the reuse flow, the
  ignored_linkages semantics, the output layout, the updated in-partition
  "found" definition of completeness, and the ./scripts/check_versions.sh
  developer invocation.

Tests (58 pass, +12 new over v1 CLI):
- Library units: multi-partition update_all_objects isolation;
  per-partition sequential accumulation; classify_linkages excludes
  wholly-external linkages and surfaces them in IgnoredLinkages.
- IO roundtrip: read_all_objects / read_partition_summaries /
  read_findable_observations round-trip (with interner re-alignment).
- CLI: sliding-partition analyze-linkages produces multi-partition
  output; --cifi-output-dir succeeds and records provenance;
  reuse manifest captures the snapshot's partition scheme (regression
  test against the reuse-defaults bug found during empirical validation);
  ignored_linkages.parquet + manifest warnings populated for sliding
  partitions; fingerprint mismatch and partition-flag conflicts
  rejected.
- Python parity tests remain byte-for-byte green.

Version: Cargo.toml 2.0.0-rc6 -> 2.0.0-rc7, pyproject.toml 2.0.0rc6 ->
2.0.0rc7 (kept in lockstep by the preflight check from PR #63).

Verified:
- cargo build --no-default-features
- cargo build --features cli
- cargo fmt -- --check
- cargo clippy --all-targets --all-features -- -D warnings
- cargo test --features cli  # 58 passed
- ./scripts/check_versions.sh  # versions consistent
- ./scripts/check_versions.sh v2.0.0rc7  # tag + versions consistent
- Empirical validation on 166M-row neomod survey with 12k synthetic linkages

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@moeyensj