Skip to content

Wasm SIMD128 beyond bitsets + amalgamation wasm differential harness#818

Closed
stereobooster wants to merge 24 commits into
RoaringBitmap:masterfrom
stereobooster:issue-801-wasm-differential-test
Closed

Wasm SIMD128 beyond bitsets + amalgamation wasm differential harness#818
stereobooster wants to merge 24 commits into
RoaringBitmap:masterfrom
stereobooster:issue-801-wasm-differential-test

Conversation

@stereobooster

@stereobooster stereobooster commented May 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Extend Wasm SIMD128 beyond src/containers/bitset.c to the rest of the hot
paths, wire it into array / bitset / run conversion code, and add a
differential-digest harness that proves native, scalar-wasm, and -msimd128
wasm produce bit-identical output.

Closes #801. Primarily a perf change.

SIMD coverage

  • array_util.c — Schlegel/Lemire-style intersect / union / xor / difference,
    optional memequals, uint16→uint32 widen, and a new
    intersect_vector16_nonempty for balanced array_container_intersect parity
    with the pcmpestra/shuffle-mask paths. SIMD vs scalar speed for the Nonempty
    variant is workload-dependent; correctness is tied to the differential
    digests, not to tools/wasm_simd_perf_smoke.sh (whole-harness wall clock).
  • bitset_util.c, bitset.c — bitset merges and SIMD bit-extract.
  • run.c — run_container_to_uint32_array uses chunked i32x4 widening
    (mirroring the AVX2 path) under __wasm_simd128__ / CROARING_WASM_SIMD,
    with no separate opt-in switch.
  • convert.c / array.c — array_container_from_bitset calls
    bitset_extract_setbits_wasm_uint16 when SIMD is enabled and the array
    capacity clears the SIMD headroom guard; otherwise scalar.

Wasm-specific helpers use a croaring_wasm_* prefix (including
croaring_wasm_array_container_to_uint32_array) to avoid colliding with the
wasm_* intrinsics in wasm_simd128.h.

Verification

  • tests/wasm_diff_harness.c — amalgamation workload that compares
    deterministic digests across native, Emscripten scalar wasm, and
    emcc -msimd128. Guarded by SIMD-uplift checks (wasm-objdump heuristic on
    the linked wasm + roaring-only objects) and an amalgamation LLVM IR trace
    for @llvm.wasm.* / @llvm.ctpop.v16i8. Temporaries are freed so leak
    sanitizers stay quiet.
  • tools/run_wasm_differential_test.sh drives the harness end-to-end.

CI / tooling

  • .github/workflows/emscripten.yml runs on ubuntu-latest via setup-emsdk
    (no Docker), and pipes through tools/check_wasm_simd_neon_pairing.sh
    (structural heuristic on Wasm/NEON elif hygiene) and
    tools/ci_wasm_differential_digest.sh.
  • CMake convenience targets: wasm_differential_test,
    wasm_neon_simd_parity_check.
  • tools/run-wasm-diff-docker.sh — optional local runner that builds a pinned
    Emscripten 3.1.74 + wabt image from an inline Dockerfile. CI does not use
    this path.
  • tools/wasm_simd_perf_smoke.sh + tools/_wasm_perf_bench_helper.py —
    optional throughput smoke. Wall-clock noise-prone; complements but does not
    replace the digest correctness story.

Misc

  • .gitignore build-debug/ and .DS_Store.

stereobooster and others added 9 commits May 24, 2026 10:05
@stereobooster @cursoragent
Add wasm differential digest test for SIMD builds.
0a8a911 
Native vs scalar wasm vs -msimd128 share one amalgamation harness and
must produce identical deterministic output; clearer diff on mismatch.

Refs RoaringBitmap#801

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Add Docker runner for wasm differential harness.
17ef52f 
Pins Emscripten 3.1.74 + WABT; only Docker needed locally.

Refs RoaringBitmap#801

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Harden wasm differential test SIMD attestation.
481fbba 
Require wasm uplift in linked binaries and roaring-only objects; scan
LLVM IR for @llvm.wasm.* intrinsic names so generic vector IR alone
cannot satisfy the SIMD leg. Skip the IR gate in CI until RoaringBitmapGH-801 SIMD
builtins land.

Refs RoaringBitmap#801

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Implement Wasm SIMD for bitsets and widen wasm SIMD verification.
3df0d00 
Bitset cardinality and bitwise container ops use wasm_simd128 when -msimd128;
differential harness and CI enforce matching digests plus SIMD bytecode uplift
and an LLVM amalgamation SIMD trace (@llvm.ctpop.v16i8 on newer Clang).

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Extend Wasm SIMD128 for array/bitset/run hot paths.
d36c446 
Adds v128 paths in array_util and bitset_util, wires containers and Neon
/simd pairing, strengthens the wasm differential harness and coverage doc,
and applies repo clang-format to touched headers and sources.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Refactor Emscripten wasm differential CI into a shell script.
82394a3 
Move digest setup (PATH, wabt install, run_wasm_differential_test) into
tools/ci_wasm_differential_digest.sh and drop the redundant wasm_simd_coverage
documentation file.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Harden WASM differential tooling and SIMD review notes.
55b9527 
Free the roaring_bitmap_or temporary in wasm_diff_harness so leak checkers stay
quiet, document WASM/x86 semantic traps and SIMD assumptions, tighten CI by
rejecting guard-skipping env in GitHub Actions, and clarify the NEON pairing
checker as structural-only.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Use croaring_wasm_* names for Wasm helpers and soften NEON pairing li…
a59f956 
…nter.

Reserve wasm_* for compiler intrinsics where possible, rename the exported array
serializer and SIMD-only internals for consistent namespacing, and document SIMD
latency expectations plus differential-test exit codes. The pairing script now
requires each Wasm #elif chain to precede Neon coverage without forcing Dummy
Wasm blocks for NEON-only work. Apply clang-format to touched headers and TU.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Drop checked-in wasm-diff Dockerfile; build image from inline pinned …
5b33f09 
…recipe.

Keeps docker-only runs for tools/run_wasm_differential_test.sh without adding
maintained Dockerfile paths under tools/docker.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster
stereobooster force-pushed the issue-801-wasm-differential-test branch from 6a9e37c to 5b33f09 Compare May 24, 2026 13:06
@stereobooster
stereobooster marked this pull request as ready for review May 24, 2026 13:30
@stereobooster @cursoragent
Rename quad pointer local in array_container_contains for Zig transla…
1a47b0f 
…te-c.

Zig 0.13's translate-c emits block labels named blk/blk_1, which collided
with the C local `blk` and produced invalid Zig; use quad_ptr instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster
stereobooster marked this pull request as draft May 24, 2026 14:37
stereobooster and others added 13 commits May 24, 2026 16:38
@stereobooster @cursoragent
Extend WASM diff harness with bounded OR-chain membership fingerprints.
18bc887 
Adds a deterministic left-associated roaring_bitmap_or fold mirroring facetsearch-style unions, validates after each inplace OR, and digests export invariants so native vs wasm SIMD parity catches phantom ids.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Add bitmap metamorphic correctness tests and widen WASM diff coverage.
1a14fa6 
CMocka bitmap_metamorphic_unit covers membership oracles for bitwise ops,
iterator/export parity, cardinality laws, inplace parity, deep OR folds,
and portable round-trip. LibFuzzer exercises OR-chain and pairwise bitmap
metamorphism. wasm_diff_harness emits matching meta_* digest lines on
native vs wasm legs; Emscripten links get larger stack/initial memory.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Apply clang-format to fuzzer, bitmap metamorphic tests, and wasm diff…
a939d31 
… harness.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Wasm SIMD paths: bitset merges, SIMD array export, and u32 SIMD broad…
2df69a5 
…cast helper.

Delegates wasm intersect_vector16 back to intersect_uint16 for digest parity against
native; SIMD bitset word ops rely on roaring_hamming after v128 bitwise combine.
Leaves bitset uint32 SIMD table decode wired out until it matches TZCNT peeling.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Fix wasm bitset SIMD extract baseVec pacing to mirror AVX2.
f786372 
The decoder must apply one base adjustment to both v128 halves of each 256-bit
table row before a single baseVec += 8; re-enable the >=8192 card fast path on
wasm32 SIMD.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Enable wasm SIMD array intersect/union/xor/difference and document el…
c4108f4 
…if pairing check.

Portable kernels are replaced with wasm128 merges and PCMPESTRI-mask emulation so
wasm32+msimd128 matches native digests end-to-end. array_container_andnot uses
difference_vector16 on wasm when outputs do not alias. README and tooling clarify
check_wasm_simd_neon_pairing.sh (runs in CI) vs run_wasm_differential_test.sh parity.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Ignore build-debug out-of-tree CMake output and macOS .DS_Store.
4784dc6 
Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Add opt-in wasm128 SIMD run_container_to_uint32_array for wasm-diff p…
734abd9 
…arity.

Exposes chunked i32x4 export mirroring AVX2; wasm amalgamation SIMD builds flip it via -DCROARING_WASM_SIMD_RUN_CONTAINER_EXPAND so digest checks stay the gate.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Use wasm SIMD uint16 bitset extract in array_container_from_bitset.
1a4ff72 
When SIMD is enabled and capacity meets the SIMD headroom check, populate the array via bitset_extract_setbits_wasm_uint16; keeps wasm-diff parity with scalar extract.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Add intersect_vector16_nonempty for balanced array_container_intersect.
e1fd172 
Uses the same Schlegel/Lemire mask path as intersect_vector16 on AVX2 and wasm SIMD, with tails delegated to intersect_uint16_nonempty.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Apply clang-format 17 across touched C/H sources.
3ea7fbc 
Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Wasm simd128 always uses SIMD run_container_to_uint32_array.
1352f13 
Drop CROARING_WASM_SIMD_RUN_CONTAINER_EXPAND—digest/CI parity is sufficient; fuzz gets rid of unreachable doc_cap guards; wasm bitset SIMD popcnt comments clarify load+scalar lowering; wasm-diff/README updated.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster @cursoragent
Document intersect_vector16_nonempty vs scalar Nonempty perf caveat.
ed6f002 
Clarify Schlegel/Lemire mask parity with intersect_vector16 paths, digest scope, and wasm_simd_perf_smoke limitations; add callsite note in array_container_intersect.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster

stereobooster commented May 24, 2026

Copy link
Copy Markdown
Author

Okay this is huge one. I can try to split it.

@stereobooster
stereobooster marked this pull request as ready for review May 24, 2026 20:04
@stereobooster @cursoragent
Apply clang-format 17 to intersect_vector16_nonempty docs.
5ed7596 
Reflow block comments so the clang-format workflow matches upstream style.

Co-authored-by: Cursor <cursoragent@cursor.com>
@stereobooster stereobooster mentioned this pull request May 24, 2026
Closed
@stereobooster
stereobooster marked this pull request as draft May 24, 2026 20:37
@lemire

lemire commented May 29, 2026

Copy link
Copy Markdown
Member

Please be careful with unmotivated changes.

Capture d’écran, le 2026-05-28 à 21 39 14

@stereobooster

stereobooster commented May 29, 2026

Copy link
Copy Markdown
Author

Please be careful with unmotivated changes.

yes this needs a break down. It is not reviewable in current shape.

Do you have any high-level feedback, regarding test-strategy choice, for example. Do not look at small details just yet

This was referenced Jun 4, 2026
Closed
Closed
Draft
@stereobooster

stereobooster commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Closing this in favor of a set of small, single-purpose PRs

Open now

Coming next, stacked on #837 (opened as each predecessor merges, to keep diffs clean and CI green):

  1. run_container_to_uint32_array + portability foundation (CROARING_WASM_SIMD)
  2. bitset containers + array_container_from_bitset
  3. array intersect/union/xor/difference
  4. intersect_vector16_nonempty

Dropped from the original branch as unrelated/unmotivated: the .DS_Store/build-debug gitignore churn, a stray roaring.hh whitespace edit, and the perf/Docker convenience tooling (wasm_simd_perf_smoke.sh, _wasm_perf_bench_helper.py, run-wasm-diff-docker.sh) — these were perf/convenience, not correctness, and can come later if wanted.

The differential-digest CI gate is added together with the first SIMD path so every PR stays green.

@stereobooster
stereobooster deleted the issue-801-wasm-differential-test branch June 4, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

WASM SIMD128 backend

2 participants

@stereobooster @lemire