Skip to content

Commit 9f9feb9

Browse files
authored
[#12954][fix] AutoDeploy: Fix Gemma4 MoE config (disable multi_stream_moe, lower free_gpu_memory_fraction) (#12955)
Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com>
1 parent d68a94e commit 9f9feb9

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

examples/auto_deploy/model_registry/configs/gemma4_moe.yaml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ max_seq_len: 8192
1616
enable_chunked_prefill: true
1717
kv_cache_config:
1818
enable_block_reuse: false
19-
free_gpu_memory_fraction: 0.8
19+
free_gpu_memory_fraction: 0.4
2020
transforms:
2121
compile_model:
2222
piecewise_enabled: true
@@ -26,5 +26,6 @@ transforms:
2626
enabled: true
2727
fuse_gemms:
2828
enabled: true
29+
#TODO(suyogg): Enable when https://github.com/NVIDIA/TensorRT-LLM/issues/12954 is resolved
2930
multi_stream_moe:
30-
enabled: true
31+
enabled: false

0 commit comments

Comments
 (0)