Update MTP inference for SpeechLM vLLM for Nemotron 3.5 Lightning based models

[DongjiGao]

Summary

• add native vLLM MTP draft-model registration and routing for NeMo SpeechLM
• support both target-only and native MTP speculative inference
• fuse SpeechLM audio embeddings into target and draft model inputs
• load target and MTP checkpoint weights, including packed MoE experts and vocabulary padding
• preserve Nemotron-H FP32 Mamba parameters and SSM cache precision
• enable PEE online inference and correct PEE audio placeholder accounting
• add focused tests for registration, configuration, weight mapping, multimodal embeddings, target-only loading, and MTP

Dependencies

This is a dependent PR based on the exact head of #16053:

• Speech Add native THD packed ASR encoders #16053: 5ab986f2bef261e36d905c20a3306fb79cb49769
• Automodel Normalization es #3555: a1df443105b2dfc413b703f623e2529f3b1d6d82
• vLLM 0.23.0

The PR targets #16053's head branch, codex/pr-16048, so its functional diff is the vLLM-plugin integration layered on that work.

The MTP implementation builds on Slyne's work in #15906 and includes the follow-up fixes required by the Nemotron Transcribe 3.5 checkpoints tested here. The Speech #16053 base predates merged fix #16054, so that weight-name mapping is present in this dependent branch as well; it can disappear when the base is refreshed from current main.

What this enables

The same exported Nemotron Transcribe checkpoint can be loaded in either mode:

# Target only
LLM(model=checkpoint, ...)

# Native MTP draft model
LLM(
    model=checkpoint,
    spec_method="mtp",
    spec_tokens=1,
    ...,
)

The target-only loader ignores checkpoint MTP-head tensors. When MTP is enabled, vLLM resolves NeMoSpeechLMMTPModel, loads those draft weights, and uses the same audio embeddings and placeholder positions as the target.

Scope

This PR contains the complete SpeechLM vLLM-plugin path for target-only and MTP inference. It deliberately contains no decoded-word repetition detector, output trimming, or other checkpoint-specific post-processing.

Validation

Focused unit suites in the project's vLLM 0.23.0 / CUDA 12.9 container with Automodel #3555 mounted:

• 156 passed
• 8 skipped
• isort 5.13.2 passed for all CI-reported Python files
• Black 24.10 passed for all changed Python files
• the repository copyright check passed locally

Real BF16 GPU smokes on one RTX PRO 6000 Blackwell Max-Q with the Hero5 Transformer-encoder checkpoint:

• target-only loaded 60.12 GiB and completed with native EOS
• native one-token MTP resolved and loaded NeMoSpeechLMMTPModel separately
• both modes produced: 10 companies that let you teach English.
• both modes produced identical token IDs and ended at EOS

The target-only and MTP smoke results, hashes, checkpoint/container provenance, and commands are retained in the Nemotron Transcribe vLLM Labbook project.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[pzelasko]

/ok to test 52d477b

[pzelasko]

/ok to test 761ea9b

[pzelasko]

/ok to test bc8de5f