Add PyTorch GPU/CPU port of phg_v2's PS4G Viterbi HMM - #187
Conversation
Ports PathFinderHMMPS4G.kt (haploid + diploid, from the phg_v2 hmm-from-ps4g-claude branch) to PyTorch: binomial emissions and gamete-switch transitions (with inbreeding-coefficient mixing for diploid) are built on the fly from a PS4G file and decoded via the existing python.hmm.viterbi Viterbi implementation, which runs on CPU or GPU. Adds a runnable CLI (impute_ps4g.py) mirroring ImputePathFromPs4g's options/output, plus unit and integration tests anchored on the Kotlin source's own test oracles. Also fixes a latent dtype bug in ps4g_io.load_ps4g_file: pandas silently inferred gameteSet/refContig as int64 whenever every PS4G row happened to have a single gamete or a purely-numeric contig name, breaking downstream string operations on legitimate PS4G files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds scripts/generate_synthetic_ps4g.py (parameterized synthetic PS4G file generator - neither repo had one) and scripts/benchmark_ps4g_hmm.py (times end-to-end CLI wall-clock across implementations/sizes/path-types, 3 repeats each), plus the resulting results CSV. Findings: Kotlin wins haploid at every size tested (cheap O(T·N) loop, so per-process startup dominates); the PyTorch port is up to ~10x faster than Kotlin for diploid at the largest tier (100K positions: 4.0s vs 40.7s median), where Kotlin's O(T·S^2) nested-loop Viterbi degrades sharply and the vectorized port doesn't. GPU is never faster than CPU in this workload - Viterbi decoding is inherently sequential (one small tensor op per position), so CUDA kernel-launch overhead never amortizes at these state-space sizes. Also fixes a real bug in impute_ps4g.py's standalone sys.path fallback (used an off-by-one `.parents[1]` instead of `.parents[2]`, so running the script directly - as the benchmark needs to, without pixi's PYTHONPATH activation - raised ModuleNotFoundError). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Speed benchmark: Kotlin vs Python-CPU vs Python-GPUAdded
Takeaways:
Also fixed a real bug found while building this: Full results: |
Standalone summary of the Kotlin vs Python-CPU vs Python-GPU benchmark (TL;DR, methodology, full results table, and reproduction steps) so the findings can be shared without needing the CSV or the live artifact link. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| else: | ||
| # Ambiguous: true gamete plus 1-2 random decoys. | ||
| n_extra = rng.randint(1, min(2, gametes - 1)) | ||
| decoys = rng.sample([g for g in range(gametes) if g != current_gamete], n_extra) |
There was a problem hiding this comment.
This strategy for choosing decoys doesn't cover the case where the true current gamete is closely related to a decoy, since decoy hits are evenly distributed and independent of previous hits. Is there a way to weight the decoy choice to simulate relatedness between gametes?
Summary
PathFinderHMMPS4G(Kotlin,hmm-from-ps4g-claudebranch) to PyTorch assrc/python/hmm/ps4g_hmm.py: haploid and diploid Viterbi path finding with binomial emissions and gamete-switch transitions (with inbreeding-coefficient F0/F1 mixing for diploid), all built on the fly from a PS4G file (docs/ps4g_specifications.md). Reuses the existing GPU-capablepython.hmm.viterbi.viterbi_decode.src/python/hmm/impute_ps4g.py, mirroring the KotlinImputePathFromPs4gcommand's options and midpoint-coordinate BED output, with a--device auto|cpu|cudaflag.ps4g_io.load_ps4g_file: pandas silently inferredgameteSet/refContigasint64whenever every row in a PS4G file happened to have a single gamete (no comma) or a purely-numeric contig name (e.g."1"), breaking downstream string operations on otherwise-valid PS4G files.tests/python/hmm/test_ps4g_hmm.py) anchored on the exact counts from the KotlinDiploidPS4GEmissionProbabilityTest.ktoracle, and integration tests (tests/python/hmm/test_impute_ps4g_cli.py) mirroringImputePathFromPs4gTest.ktscenarios (single-gamete calls, exact BED midpoint coordinates, recombination switching, inbred-homozygous diploid,n-parentsrestriction, CLI validation errors).Test plan
pixi run -- python -m pytest tests/python/hmm -q— 38/38 passed, including GPU-vs-CPU path-parity test (ran for real on an H200 in this environment, not skipped).pixi run -- python -m pytest tests/python -q(excluding pre-existing, unrelated broken modules:mamba_ssmnot installed, GLIBCXX mismatch for scipy/numba) — confirmed identical pre-existing failure set before/after this change (0 regressions).--device cpuand--device cudaproduce byte-identical output.