BUG: main(1dd0a7889901097555262d3ff3521018a2e3ece8): Fatal Python error: Segmentation fault

[Honghe]

sglang-log-20260409.txt

Reminder

• I have read the above rules and searched the existing issues.

System Info

(ktransformers) ubuntu@ESC8000A-E11:~$ lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             48 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      256
  On-line CPU(s) list:       0-255
Vendor ID:                   AuthenticAMD
  Model name:                AMD EPYC 7B13 64-Core Processor
    CPU family:              25
    Model:                   1
    Thread(s) per core:      2
    Core(s) per socket:      64
    Socket(s):               2
    Stepping:                1
    Frequency boost:         enabled
    CPU(s) scaling MHz:      43%
    CPU max MHz:             3541.3679
    CPU min MHz:             1500.0000
    BogoMIPS:                4491.45
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse s
                             se2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpu
                             id extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic
                             movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a mis
                             alignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l
                             lc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep
                              bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xs
                             aves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdp
                             ru wbnoinvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid dec
                             odeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmu
                             lqdq rdpid overflow_recov succor smca fsrm debug_swap ibpb_exit_to_user
Virtualization features:
  Virtualization:            AMD-V
Caches (sum of all):
  L1d:                       4 MiB (128 instances)
  L1i:                       4 MiB (128 instances)
  L2:                        64 MiB (128 instances)
  L3:                        512 MiB (16 instances)
NUMA:
  NUMA node(s):              2
  NUMA node0 CPU(s):         0-63,128-191
  NUMA node1 CPU(s):         64-127,192-255
Vulnerabilities:
  Gather data sampling:      Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Mitigation; Safe RET
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS N
                             ot affected; BHI Not affected
  Srbds:                     Not affected
  Tsa:                       Vulnerable: Clear CPU buffers attempted, no microcode
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace

(ktransformers) ubuntu@ESC8000A-E11:~$ nvidia-smi
Thu Apr  9 14:30:37 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 35%   31C    P8             30W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:61:00.0 Off |                  Off |
| 35%   28C    P8              7W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 4090        Off |   00000000:81:00.0 Off |                  Off |
| 35%   33C    P8             13W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 4090        Off |   00000000:E1:00.0 Off |                  Off |
| 35%   32C    P8             13W /  450W |       4MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |

Reproduction

The following configuration worked properly in the previous version.

(ktransformers) ubuntu@ESC8000A-E11:~$ SGLANG_USE_MODELSCOPE=true \
python -m sglang.launch_server \
--host 0.0.0.0 \
--port 8000 \
--model /home/ubuntu/.cache/modelscope/hub/models/Qwen/Qwen3___5-122B-A10B-FP8 \
--kt-weight-path /home/ubuntu/.cache/modelscope/hub/models/Qwen/Qwen3___5-122B-A10B-FP8 \
--kt-cpuinfer 124 \
--kt-threadpool-count 2 \
--kt-gpu-experts-ratio 0.62 \
--kt-method FP8 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.9 \
--chunked-prefill-size 8192 \
--max-running-requests 1 \
--max-mamba-cache-size 4 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 128000 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--enable-mixed-chunk \
--tensor-parallel-size 4 \
--enable-p2p-check \
--disable-shared-experts-fusion \
--disable-custom-all-reduce \
--kt-gpu-prefill-token-threshold 2048

[sglang-log-20260409.txt](https://github.com/user-attachments/files/26591184/sglang-log-20260409.txt)

[2026-04-09 14:28:07 TP0] max_total_num_tokens=123557, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=1, context_len=262144, available_gpu_mem=2.11 GB
[2026-04-09 14:28:08] INFO:     Started server process [140632]
[2026-04-09 14:28:08] INFO:     Waiting for application startup.
[2026-04-09 14:28:08] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-04-09 14:28:08] INFO:     Application startup complete.
[2026-04-09 14:28:08] INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2026-04-09 14:28:09] INFO:     127.0.0.1:53654 - "GET /model_info HTTP/1.1" 200 OK
[2026-04-09 14:28:11 TP0] Prefill batch, #new-seq: 1, #new-token: 80, #cached-token: 0, full token usage: 0.00, mamba usage: 0.50, #running-req: 0, #queue-req: 0, input throughput (token/s): 0.00, cuda graph: False
Fatal Python error: Segmentation fault

Thread 0x0000721c47fff6c0 (most recent call first):
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 145 in _watchdog_once
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 125 in _watchdog_thread
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x0000721c63fff6c0 (most recent call first):
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 73 in _recv_msg
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 228 in _read_thread
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x0000721c6dfff6c0 (most recent call first):
  File "/usr/lib/python3.12/threading.py", line 359 in wait
  File "/usr/lib/python3.12/threading.py", line 655 in wait
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x00007239b5fff6c0 (most recent call first):
  File "/usr/lib/python3.12/threading.py", line 359 in wait
  File "/usr/lib/python3.12/threading.py", line 655 in wait
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x0000726c12f75140 (most recent call first):
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/cuda/graphs.py", line 141 in replay
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 1063 in replay
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2459 in _forward_raw
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2394 in forward
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 456 in forward_batch_generation
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2357 in run_batch
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 1118 in event_loop_normal
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120 in decorate_context
  File "/home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3159 in run_scheduler_process
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
  File "/usr/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pybase64._pybase64, charset_normalizer.md, charset_normalizer.cd, requests.packages.charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex, markupsafe._speedups, cuda_utils, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.option, av.descriptor, av.format, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._batched_linalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, grpc._cython.cygrpc, google._upb._message, cuda.cudart, cuda.nvrtc, __triton_launcher, uvloop.loop (total: 162)
!!!!!!! Segfault encountered !!!!!!!
  File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000726c12c4532f
  File "./nptl/pthread_kill.c", line 44, in __pthread_kill_implementation
  File "./nptl/pthread_kill.c", line 78, in __pthread_kill_internal
  File "./nptl/pthread_kill.c", line 89, in __GI___pthread_kill
  File "../sysdeps/posix/raise.c", line 26, in __GI_raise
  File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000726c12c4532f
  File "./nptl/pthread_mutex_lock.c", line 80, in ___pthread_mutex_lock
  File "./nptl/pthread_create.c", line 447, in start_thread
  File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 78, in clone3
  File "<unknown>", line 0, in 0xffffffffffffffff

[rank2]:[W409 14:28:14.357604258 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=72, addr=[localhost]:55578, remote=[localhost]:38049): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:682 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x80 (0x77803977cb80 in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ffc5b1 (0x7780231fc5b1 in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5ffd9ad (0x7780231fd9ad in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5ffe55a (0x7780231fe55a in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x31e (0x7780231f927e in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x777fe1c55868 in /home/ubuntu/ktransformers/.venv/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xecdb4 (0x7781c1cecdb4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x9caa4 (0x7781c509caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x129c6c (0x7781c5129c6c in /lib/x86_64-linux-gnu/libc.so.6)

Others

No response

[skyguan92]

I checked whether PR #1918 could be the root cause here, since current main (1dd0a7889901097555262d3ff3521018a2e3ece8) is the merge commit of that PR.

I ran an A/B check between:

• parent: 9b2d3b687bc8cdc25843dceba173a82109695554
• merge: 1dd0a7889901097555262d3ff3521018a2e3ece8

Test environment:

• host: x86_64
• CPU: Intel Xeon Platinum 8488C
• GPUs: 2x RTX 4090
• Python env: existing kt env on that machine

What I tested:

• rebuilt kt-kernel from source at both commits
• ran a focused stress test on the exact path touched by kt-kernel: enable CPUInfer stream bridge for ROCm #1918:
  CPUInfer.submit_with_cuda_stream(...) / CPUInfer.sync_with_cuda_stream(...)
• used kt-kernel/examples/bench_moe_amx_int8.py --use_cuda_stream as the minimal reproducer for that bridge path

Notes:

• my first run with a larger config OOM'ed on both commits because both 4090s were already occupied by VLLM::Worker_TP*, so that result was not useful
• I then reduced the benchmark config to fit the remaining GPU memory and reran it

Stable repro command used for the A/B check:

CUDA_VISIBLE_DEVICES=0 python bench_moe_amx_int8.py \
  --use_cuda_stream \
  --layer_num 1 \
  --expert_num 8 \
  --num_experts_per_tok 2 \
  --hidden_size 256 \
  --intermediate_size 128 \
  --workload 8 \
  --warmup_iter 20 \
  --test_iter 2000 \
  --cpuinfer_threads 8 \
  --numa_count 1

I also ran a longer stress pass with --test_iter 20000.

Results:

• parent commit: passed, no segfault
• merge commit: passed, no segfault
• 20k-iteration run:
  • parent: 1664.92 us/iter
  • merge: 1665.27 us/iter

So at least from this A/B, I could not reproduce any regression introduced by #1918 on the CUDA stream bridge path itself.

Important boundary of this result:

• this is not a full AVX2 + GPTQ INT4 reproduction
• the available machine was Intel AVX512/AMX, not a pure AVX2 host
• this only isolates the code path actually changed by kt-kernel: enable CPUInfer stream bridge for ROCm #1918, not the whole higher-level prefill / GPTQ / scheduler stack involved in this issue

So current evidence does not support #1918 as the direct root cause of this segfault. If someone can provide a true AVX2 + GPTQ INT4 repro or a bisect that points into #1918 specifically, I can keep digging from there.

[Honghe]

@skyguan92 the model and --kt-method is FP8, not GPTQ-INT4.

[yyj6666667]

@Honghe
I tried to reproduce on the latest main with AVX2 + RTX 5090 but couldn't hit the segfault on my side, so bisecting locally isn't really working for me.

When you have some time, would you mind running a few extra builds to help pin this down — ideally the two closest commits you can find (one last-known-good, one first-known-bad), and maybe a couple of points in between if possible? Even a rough pair would help narrow the window significantly.

Thanks for bearing with us on this!

[usrlocalben]

I have observed this problem (or something that gives identical-at-a-glance log output, notably the fail point being right after the prefill result log line) recently while troubleshooting other issues with a lot of rebuild churn between checkout versions.
I'm suspicious that it was due to ABI mismatch somewhere. At some point, rebuilding kt-kernel gave me a working system again. If I can pinpoint trouble I'll update here, but I suggest you re-check your worktree, kt/sg commit consistency, and use a clean build etc.

edit: this was on AVX512.