AMD PR Set 2 of 6 — Review and Summarization of MMQ and MoE - MatMul Optimizations

[Geramy]

AMD PR Set 2 of 6 — Review and Summarization of MMQ and MoE - MatMul Optimizations

Johannes and llama.cpp team, below is a group of related patches from AMD intended to optimize a set of high-priority models on the ROCm backend. I've grouped these together — from a larger set of patches — to hopefully make your job easier in reviewing them, and I've provided a short summary for each. These PR numbers (#32, #39, etc.) refer to the AMD team's downstream fork at AMD-Ecosystem/llama.cpp, where this work landed first; the goal is to bring it upstream in reviewable batches.

This second group is the MMQ path: kernel rewrites, MoE-aware tile sizing, and an decode-path matvec.

2. MMQ / MoE Matmul Optimizations

1. #32 — RDNA3.5 (gfx11 / gfx1151) MMQ prefill optimizations — liangliangchang
   • MMQ kernel rewrites tuned for occupancy and tile sizing. Reworks Q4_K, Q6_K, dequant, vec_dot, and the tiny-M path, and documents the HIP/clang codegen workarounds it relies on.
2. #39 — ggml-cuda: size MMQ tile to MoE tokens-per-expert — roberteg16
   • For MUL_MAT_ID, picks tile width from ~2x the average tokens-per-expert instead of the worst case (all tokens on one expert). RDNA3.5 only; dense and decode untouched. +91% pp128, +39% pp1024 on Qwen3.6-35B-A3B.
3. #63 — mmq: compacted MoE tiling for RDNA3.5 (default on) — roberteg16
   • Launches only the minimum number of column-blocks implied by the actual per-expert token counts, instead of the worst-case grid. Modeled on vLLM's moe_align_block_size. Was opt-in; now default on, with the env flag removed.
4. #26301 — ggml-cuda: add dequant-float matvec (mmvdq) for Q4_K/Q5_K/Q6_K on RDNA3.5 — Annieren
   • Decode-path (n=1) matvec that skips the q8_1 activation quantize pass — pure overhead when bandwidth-bound — and dequantizes weights straight to float. Includes a fused gate+up SwiGLU variant. Default on for RDNA3.5 only.
5. #25 — Add mmq device table for RDNA3.5 — Annieren
   • Sets two MMQ constants for RDNA3.5, matched host and device: tile height mmq_y 128 -> 64, warps-per-block 8 -> 4. Measured across 27 dense + MoE models on gfx1151: prefill +14–18% at n=128, +2–8% at 512–4096, decode neutral, PPL bit-identical.
6. #62 — tests: add MoE MMQ benchmark with routing-distribution generator — roberteg16
   • Standalone benchmark-moe-mmq for the mul_mat_id kernel. Generates controlled routing distributions (uniform / single / zipf / concentration) and reports a roofline per row. Already based on master.

Note: Happy to do the front-line review on these conventions, obvious issues, and benchmark coverage.

Discussions:
AMD PR Set 1 of 6 — Tooling and Profiling modifications and additions
AMD PR Set 2 of 6 — MMQ and MoE - MatMul Optimizations
AMD PR Set 3 of 6 — Flash Attention Optimization RDNA3.5
AMD PR Set 4 of 6 Matvec and Fusion Decode Optimizations
AMD PR Set 5 of 6 — GEMM Prefill Weight Optimizations
AMD PR Set 6 of 6 - Layout and Concurrency Optimizations

[Geramy]

@am17an

[jeremyfowers]

Awesome, thanks @Geramy. @am17an - I will keep working on getting you a hardware shipment ASAP.

[jimw567]

@Geramy and @am17an Regarding hardware, I have just secured two boards (strix halo and strix point) to be used as GHA runners in the public domain. Can you please let me know who I can work with to add them up to the list of runners in upstream llama.cpp?

[am17an]

The correct person maybe @taronaeo

[taronaeo]

@jimw567 Thanks for emailing me. I'll collaborate with you there.

[Geramy]

#26378 (reply in thread) @jimw567

[jimw567]

Regarding item 4: "Decode-path (n=1) matvec that skips the q8_1 activation quantize pass — pure overhead when bandwidth-bound — and dequantizes weights straight to float. Includes a fused gate+up SwiGLU variant. Default on for RDNA3.5 only.", Annie has already raised an upstream PR below:

#26301

[Geramy]

I'll update the discussion

[jimw567]

#26301 has been in wait for review state for quite a few days. What is the general timeline that a PR will be picked up by a maintainer and reviewed?

[am17an]

@jimw567 most HIP maintainers are volunteers and don't exactly have a lot of bandwidth at the moment. Patience is advised

[jimw567]

@am17an Thanks very much for your reply! Sure, I am new to this project and am still learning the process. I have started monitoring the repo and will contribute wherever I can.

[liangliangchang]

For AMD-Ecosystem#32, we plan to split it to several sub-PRs to make it easier to review. I just opened this PR #28195 as the beginning. Could you please review it? Thanks!