Skip to content

fix: handle non-contiguous tensors in IPC weight refit#2418

Merged
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
jlcanta:fix/pack-tensor-non-contiguous
May 28, 2026
Merged

fix: handle non-contiguous tensors in IPC weight refit#2418
terrykong merged 1 commit into
NVIDIA-NeMo:mainfrom
jlcanta:fix/pack-tensor-non-contiguous

Conversation

@jlcanta

@jlcanta jlcanta commented May 5, 2026

Copy link
Copy Markdown
Contributor

What does this PR do ?

Make pack_tensor and packed_broadcast_producer accept non-contiguous tensors yielded by the params iterator, instead of crashing with RuntimeError: view size is not compatible with input tensor's size and stride.

Issues

Fixes #2417

Usage

No user-facing API change. Internal helpers (stream_weights_via_ipc_zmq_impl.pack_tensor and packed_broadcast_producer) become robust to non-contiguous inputs:

# before: RuntimeError
tensor.data.view(-1).view(dtype=torch.uint8)

# after: works whether the tensor is contiguous or not
tensor.data.reshape(-1).view(dtype=torch.uint8)

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests? → tests/unit/models/policy/test_utils.py::TestStreamWeightsViaIPC::test_stream_weights_via_ipc_zmq_impl_non_contiguous
  • Did you run the unit tests and functional tests locally? → ran ruff check and ruff format --check on the touched files; the unit test requires CUDA which I don't have locally
  • Did you add or update any necessary documentation? → no docs change needed (internal helper, behaviour is now strictly more permissive)

Additional Information

Root cause. stream_weights_via_ipc_zmq_impl consumes tensors from megatron_bridge.export_hf_weights. For the Qwen3ForCausalLM converter, QKVMapping.megatron_to_hf → split_qkv_weights finishes with q.reshape(-1, hidden_size) (and same for k, v). Tensor.reshape is allowed to return a view with strides that are compatible with reshape but not with view. The next step in NeMo-RL was tensor.data.view(-1), which raises.

Fix.

  1. nemo_rl/models/policy/utils.pypack_tensor now uses reshape(-1) instead of view(-1). reshape returns the same view in the contiguous case (zero copy) and falls back to a copy when not — and since the next operation is .copy_() into a buffer, the extra copy in the rare non-contiguous case has no extra cost.
  2. nemo_rl/utils/packed_tensor.pypacked_broadcast_producer adds .contiguous() before .view(torch.uint8).view(-1) (defense in depth — same root cause).
  3. tests/unit/models/policy/test_utils.py — refactored the existing IPC roundtrip test into a reusable helper _run_stream_weights_roundtrip, and added test_stream_weights_via_ipc_zmq_impl_non_contiguous that exercises the helper with three kinds of non-contiguous inputs (transpose, strided slicing, permute). Without the fix, these new inputs trigger the original RuntimeError.

Tested manually.

Model converter_type Result before fix Result after fix
Qwen/Qwen3-1.7B Qwen2ForCausalLM refit works refit works
Qwen/Qwen3.5-2B Qwen3ForCausalLM crashes in pack_tensor refit works

Compatibility. Strictly additive — when the input tensor is already contiguous (the common path) the behaviour is identical to before.

@jlcanta
jlcanta requested review from a team as code owners May 5, 2026 21:43
@copy-pr-bot

copy-pr-bot Bot commented May 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label May 8, 2026
@youngeunkwon0405 youngeunkwon0405 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label May 8, 2026
@youngeunkwon0405

Copy link
Copy Markdown
Contributor

/ok to test 8384171

@youngeunkwon0405 youngeunkwon0405 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@terrykong, I think I had a similar issue when I tested the qwen3.5 model, and the fix was pretty much similar to this one at that time.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-customer Waiting on the original author to respond waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond waiting-on-customer Waiting on the original author to respond labels May 8, 2026
pack_tensor in stream_weights_via_ipc_zmq_impl and packed_broadcast_producer
called view(-1) on tensors yielded by megatron_bridge.export_hf_weights,
which can be non-contiguous for the Qwen3ForCausalLM converter (the bridge's
split_qkv_weights finishes with reshape(-1, hidden_size), and reshape may
return a view with strides that are valid for reshape but not view).

Use reshape(-1) in pack_tensor, and add .contiguous() before
view(torch.uint8).view(-1) in packed_broadcast_producer. Both are no-ops in
the contiguous case. Add a regression test exercising the IPC roundtrip
with transposed, strided-slice, and permuted tensors.

Fixes NVIDIA-NeMo#2417

Signed-off-by: Jose Luis <jlcanta02@gmail.com>
@youngeunkwon0405
youngeunkwon0405 force-pushed the fix/pack-tensor-non-contiguous branch from 8384171 to e1d8e95 Compare May 22, 2026 00:36
@youngeunkwon0405

Copy link
Copy Markdown
Contributor

/okay to test e1d8e95

@youngeunkwon0405

Copy link
Copy Markdown
Contributor

@terrykong, I believe, to run QWEN3.5 we need this fix. It is almost one line change, could you please help reviewing?

@terrykong
terrykong merged commit 684080e into NVIDIA-NeMo:main May 28, 2026
37 checks passed
@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-maintainers Waiting on maintainers to respond label May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) community-request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pack_tensor crashes on non-contiguous tensors during Megatron→vLLM refit (Qwen3.5 / Qwen3ForCausalLM converter)

4 participants