[CUDA] Qwen3 Model quality regression

[jixiongdeng]

Describe the issue

Qwen3 0.6B models generate gibberish outputs on CUDA backend on main branch.
After investigation, the issue was introduced from an earlier commit: 39d8520b9bfe16217d56be38293fc82556d67b5b
It is reproducible on both unquantized model(fp16) and rtn 4bit model.

#fp16
Prompt: 1+1=?

1+1+1+1+10+11+100

systemsystemsystem


#u4 rtn
Prompt: 1+1=?

1+1+1+1+1+1+1+1+1+1+1+1=?

Let? 1+1+11+1+1+1+1+1. Let 1+1+1+1+11+1+11+11+1+11+1+1+1+1+1+1+1+1+1+1+1+1+

The last work commit is: 5b88e4ed6778615113ad4831b160ec52e0d5ac71

To reproduce

Models are generated from onnxruntime-genai's model builder. fp16 and int4.

I built onnxruntime-gpu with following command:

./build.sh --config Release --build_wheel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/local/cuda --parallel --skip_tests

pip install onnxruntime/build/Linux/Release/dist/onnxruntime_gpu-*.whl --force-reinstall

Prompt&Answers are generated with og.generator.

Urgency

This commit is included in ORT1.24 release. Qwen3 are popular open source models.

Platform

Linux

OS Version

6.6.117.1-1.azl3

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1817f4a

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.8

[apsonawane]

Hello Jixiong Deng (@jixiongdeng), what version of onnxruntime-genai are you using?

[titaiwangms]

I can repro from main. Not the exact output, but it does output something weird, while 1.23.2 ort output fine and perfectly.

I confirm the commit 5b88e4e working. But the commit 36017ad before 1817f4a does not work already, so I am not sure if the PR is the cause though (also I don't see it's included in rel-1.24.0).

I can try the current rel-1.24.0 and see if the issue exists in the branch.

[titaiwangms]

rel-1.24.0 also repro the issue.

[titaiwangms]

I confirmed Tianlei Wu (@tianleiwu) has the fix in https://github.com/microsoft/onnxruntime/commits/tlwu/gqa_xqa/