Environment
- Hardware: NVIDIA Jetson Orin Nano Super (SM87, 8GB LPDDR5 unified memory, 7619 MiB reported by CUDA)
- L4T: R36.4.7, kernel 5.15.148-tegra
- CUDA: 12.6
- Model: gemma4-e2b-IQ2_M.gguf (Gemma 4 E2B)
- Build:
cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DCMAKE_BUILD_TYPE=Release
- Branch:
feature/turboquant-kv-cache (e93b7c5)
Flags
--cache-type-k turbo4 --cache-type-v f16 -ngl 99 -c 16384 --no-mmap --no-warmup -b 128 -ub 128
Behaviour
Server starts and handles short prompts correctly (<2K tokens). At ~8K+ tokens the server crashes silently — no stderr output, no log entries, process just disappears.
Short context (~43 tokens): works, 23.5 tok/s decode
Medium context (~1.5K tokens): works, 24.4 tok/s decode
Long context (~4K tokens): works, 23.5 tok/s decode
~8K tokens: silent crash
Control
Stock llama.cpp llama-server (same model, same flags minus turbo4) handles 8K+ without issue, ruling out OOM or L4T allocation bug.
Suspected cause
The silent crash with no error output suggests a hard CUDA abort in the kernel — likely the warp-cooperative 512-WHT write path or a syncthreads deadlock in the D=512 SET_ROWS path under higher token counts. The document describing this implementation mentions both a syncthreads deadlock fix (warps exiting j-loop early) and a warp reduction deadlock (only 8/32 threads participating in shfl_down). These may not be fully resolved for SM87 at longer sequences.
Notes
- CUDA OOM is ruled out — 7619 MiB unified memory, model only uses ~1150 MiB, plenty of headroom
- turbo3/turbo2 not yet tested (may exhibit same issue)
- Crash occurs on fresh boot with jetson_clocks enabled
Environment
cmake -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=87 -DCMAKE_BUILD_TYPE=Releasefeature/turboquant-kv-cache(e93b7c5)Flags
Behaviour
Server starts and handles short prompts correctly (<2K tokens). At ~8K+ tokens the server crashes silently — no stderr output, no log entries, process just disappears.
Short context (~43 tokens): works, 23.5 tok/s decode
Medium context (~1.5K tokens): works, 24.4 tok/s decode
Long context (~4K tokens): works, 23.5 tok/s decode
~8K tokens: silent crash
Control
Stock llama.cpp
llama-server(same model, same flags minus turbo4) handles 8K+ without issue, ruling out OOM or L4T allocation bug.Suspected cause
The silent crash with no error output suggests a hard CUDA abort in the kernel — likely the warp-cooperative 512-WHT write path or a syncthreads deadlock in the D=512 SET_ROWS path under higher token counts. The document describing this implementation mentions both a syncthreads deadlock fix (warps exiting j-loop early) and a warp reduction deadlock (only 8/32 threads participating in shfl_down). These may not be fully resolved for SM87 at longer sequences.
Notes