Skip to content

MiniMax-M3-NVFP4 B200 recipe: validation numbers predate the NVFP4 correctness fix (vllm#48929); offer to re-validate + PR post-fix numbers #764

Description

@stewtong

The MiniMax-M3-NVFP4 B200 recipe's "validated benchmark" note (#605, merged 2026-07-09) predates the NVFP4 inference-correctness fix vllm-project/vllm#48929 (merged 2026-08-05). No stable release contains that fix (v0.26.0 was cut 2026-07-27), so the page's validation numbers were measured on a build carrying the pre-fix swiglu/indexer behavior.

We re-ran the recipe on a post-fix nightly (0.26.1rc1.dev457+gc810e5ee9) on 8x B200 with correctness gates in front of every measurement. Full report with methodology, raw-data conventions, and startup logs: vllm-project/vllm#51494. Highlights on the post-fix build:

Measurement Result
Correctness (temp 0, content-checked 7-prompt suite) 7/7 pass; needle recovery 144/144 (envelope) + 15/15 (ladder, 128K-1M real prose)
1M real-prose TTFT, c=1 43.2s p50 (1,011,492 input tokens, 3 reps)
Aggregate prefill throughput ~54K input tok/s at 128K c=16
EAGLE3 (Inferact/MiniMax-M3-EAGLE3, 3 spec tokens) 2.1-2.3x decode-window gain at +5% TTFT (c=1)

Two proposals for the recipe page:

  1. Re-validate on a post-#48929 build and state a minimum build date. We are happy to PR the post-fix numbers above directly, including the reproduction commands.
  2. Add a note on the nightly window 2026-08-05 to 2026-08-07, where images fail at startup (TYPE_UNSTABLE_JOIN in the SM100 FMHA path, quack-kernels 0.6.3 under cutlass-dsl 4.6; fixed upstream by [Bugfix] Pin quack-kernels before FA4 AST regression vllm#51418). Anyone following the recipe on a nightly from that window needs quack-kernels<0.6.3.

Optional third item, since the data supports it: given the 2.1-2.3x decode gain at +5% TTFT and the EAGLE3 config's higher KV capacity (2,727,296 vs 2,558,976 tokens, from --language-model-only freeing the vision tower), the recipe could present EAGLE3 as the recommended pairing rather than an option. Details and acceptance counters in vllm-project/vllm#51494.

If maintainers agree on direction, tell us which shape you want and we'll open the PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions