[MLAS] Adding DynamicQGemm function pointers and ukernel interface - #27403
Merged
Hariharan Seshadri (hariharans29) merged 1 commit intoFeb 21, 2026
Merged
Conversation
Contributor
Author
|
@microsoft-github-policy-service agree company="Arm" |
Member
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 4 pipeline(s). |
Member
|
Also fixes related issue #26361 |
Hariharan Seshadri (hariharans29)
approved these changes
Feb 20, 2026
Hariharan Seshadri (hariharans29)
enabled auto-merge (squash)
February 20, 2026 23:12
Hariharan Seshadri (hariharans29)
merged commit Feb 21, 2026
df9146f
into
microsoft:main
88 checks passed
Tianlei Wu (tianleiwu)
added a commit
that referenced
this pull request
Feb 27, 2026
This cherry-picks the following commits for the release: | Commit ID | PR Number | Commit Title | |-----------|-----------|-------------| | decd177 | #27090 | Fix GatherND division by zero when batch dimensions mismatch | | 55f8234 | #27360 | Fix QMoE CPU Operator | | df9146f | #27403 | [MLAS] Adding DynamicQGemm function pointers and ukernel interface | | 0f93853 | #27318 | [js/web] Use embedded WASM module in Blob URL workers when wasmBinary is provided | | b2a6e69 | #27364 | QMoE CPU Performance Update (Up to 4x on 4-bit) | | f501e1d | #27413 | Fix refcount bug in map input conversion that caused shutdown segfault | | b32b205 | #27421 | Fix error where bytes is not assigned for dynamic qgemm pack b size | | 426b006 | #27397 | Fix DllImportResolver | | 0982844 | #27412 | MatmulNBits prepacking scales fix | | 9afb0d2 | #27430 | Fix validation for external data paths for models loaded from bytes | | 71d2cd0 | #27401 | Enable Python 3.14 CI and Upgrade Dependencies | | 79e0676 | #27419 | fix: out of bounds access for resize operation | | 82eb99c | #27459 | Fix SkipLayerNorm fusion incorrectly applied when gamma/beta are not 1D | | 355278a | #27444 | Fix GatherCopyData Integer Truncation Leading to Heap Out-of-Bounds Read/Write | | cf96123 | #27411 | [web] fix usage of wasmBinary together with a blob URL for .mjs | | 1131a86 | #27399 | [web] remove the unhelpful "Unknown CPU vendor" warning. | | ffbbc4f | #27316 | Build Windows ARM64X binaries as part of packaging pipeline | --------- Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: patryk-kaiser-ARM <patryk.kaiser@arm.com> Co-authored-by: don <70039285+0-don@users.noreply.github.com> Co-authored-by: Jonathan Clohessy <jonathan.clohessy@arm.com> Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Adrian Lizarraga <adlizarraga@microsoft.com> Co-authored-by: Lukas Folle <126877803+lukas-folle-snkeos@users.noreply.github.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: Yulong Wang <7679871+fs-eire@users.noreply.github.com> Co-authored-by: Chaya <cha182350@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: Erik <erscor@microsoft.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
This was referenced Mar 9, 2026
This was referenced Mar 17, 2026
This was referenced Mar 30, 2026
This was referenced Apr 20, 2026
This was referenced May 1, 2026
This was referenced May 8, 2026
This was referenced Jul 1, 2026
This was referenced Jul 8, 2026
This was referenced Jul 22, 2026
This was referenced Aug 1, 2026
This was referenced Aug 8, 2026
This was referenced Aug 17, 2026
Kanishk Pachauri (Mr-Sunglasses)
added a commit
to VocaHQ/vocaphone
that referenced
this pull request
Aug 19, 2026
The OnePlus 15 log ends the way the exit reporting was added to catch: a Parakeet dictation reaches local_transcription_started, the state goes TRANSCRIBING, and 700 ms later the process is gone with signal=sigill. It happens with a sherpa model and not with whisper.cpp on the same phone, which is the whole diagnosis in one line -- the two engines share nothing but the microphone. ONNX Runtime 1.23.2, which is what we shipped, installs the KleidiAI GEMM and convolution overrides whenever MLAS_CPUIDINFO::HasArm_SME() is true. Every ukernel behind those overrides is an SME2 one, and there is no second check: at that tag sgemm_kleidiai.cpp mentions SME2 zero times, and disassembling the library we shipped agrees -- its MLASCPUIDInfo reads HWCAP bit 20 and HWCAP2 bits 9, 13 and 14, and nothing else. SME and SME2 are not the same extension, and asking only about the first is how you end up running the second. The OnePlus 15's SM8850 is the first Arm chip we ship to that implements SME without SME2. Its /proc/cpuinfo lists sme, smei8i32, smef16f32, smeb16f32 and smef32f32, and no sme2, so the first fp32 matmul in the Parakeet encoder is an opcode the core does not have. Whisper never reaches this code: ggml picks its CPU backend from the features it actually finds, which is what the seven libggml-cpu-android_armv*.so variants in the APK are for, and it never goes through MLAS. Upstream is microsoft/onnxruntime#26377, where #26678 is this crash on this phone at this version, fixed by microsoft/onnxruntime#27403. 1.28.0 is the first release clear of that and of a second SM8850 bug in 1.27.x that miscomputes zipformer encoders with no error at all (k2-fsa/sherpa-onnx#3845). So this rebuilds sherpa-onnx v1.13.6 against 1.28.0 rather than taking its published Android release, which pins 1.27.1. In the new library the two features are separate booleans, HWCAP2 bit 23 for SME and bit 37 for SME2, and there is an mlas.disable_kleidiai session option to fall back on if anything else surfaces. The two libraries have to move together. libsherpa-onnx-jni.so imports exactly one symbol from ONNX Runtime and it carries a version tag that changes every release, OrtGetApiBase@VERS_1.23.2 before and @VERS_1.28.0 now, so replacing only the runtime would fail to load rather than fail to work. The five Kotlin files under com/k2fsa/sherpa/onnx come from the same tag for the same reason: the JNI resolves their config fields by name, and v1.13.6 added qnnConfig, hotwords and cohereTranscribe. jniLibs/README.md records the versions, the archive hash and the build, which is the upstream script with SHERPA_ONNX_ONNXRUNTIME_ROOT pointed at 1.28.0 and the same max-page-size flag the whisper libraries get. just ci passes -- 393 tests, lint clean, and check-page-alignment.py holds for all four new libraries. testFdroidDebugUnitTest passes too, and the release APK builds through R8 and lintVital. arm64-v8a grows by 2.8 MB and armeabi-v7a by 1.3 MB, all of it ONNX Runtime. This is the crash the previous commit could only describe. It needs confirming on the OnePlus 15 itself, since no device here has SME at all.
This was referenced Aug 21, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
Fixes #26377