[Performance] ORT 1.29 CPU FP16 Gemm is ~3500x slower than 1.28 on Windows x64

[withsalt]

Describe the issue

Upgrading Microsoft.ML.OnnxRuntime from 1.28.0 to 1.29.0 causes a severe
performance regression for FP16 Gemm on the CPU Execution Provider on
Windows x64.

A minimal single-node FP16 Gemm model with:

• A: [14, 1024]
• B: [4096, 1024], transB=1
• Y: [14, 4096]

produces the following warmed-up mean latency:

Version	Mean latency
ONNX Runtime 1.28.0	0.121 ms
ONNX Runtime 1.29.0	430.339 ms

This is approximately a 3,556x regression.

ORT profiling shows that the execution path changed:

• ORT 1.28 promotes the FP16 Gemm subgraph to FP32 through
  InsertCastTransformer; the Gemm kernel receives float tensors.
• ORT 1.29 executes the node directly using the new CPU MLFloat16 Gemm
  kernel; the Gemm kernel receives float16 tensors.

The regression is also visible in a real Qwen3-ASR FP16 encoder containing
146 Gemm nodes:

Metric	ORT 1.28	ORT 1.29
146 Gemm nodes, total	24.018 ms	19,338.133 ms
Median Gemm node	0.098 ms	98.738 ms
Encoder, 5.69 s input	201 ms	120,620 ms
End-to-end inference	2,649 ms	139,762 ms

In the ORT 1.29 encoder profile, FP16 Gemm accounts for approximately 99.4%
of total encoder execution time.

This appears related to #29709, which registered native FP16 MatMul/Gemm/Conv
kernels for the CPU EP. On Windows x64 without an accelerated FP16 Gemm
backend, execution appears to fall back to the portable/Eigen FP16 path.
Because a matching FP16 kernel now exists, the previous FP32 promotion no
longer occurs.

This differs from #32186/#32197: the FP16 kernel exists and the session loads,
but the selected kernel is dramatically slower than the previous FP32
fallback.

Expected behavior:

• Preserve the FP32 promotion on CPUs without an accelerated FP16 Gemm
  backend; or
• Use a cost model/shape heuristic before retaining FP16 nodes; or
• Expose a SessionOption that disables CPU FP16 Gemm/MatMul kernels.

Pinning ONNX Runtime to 1.28.0 is currently the only reliable workaround.

To reproduce

1. Generate the minimal model:

import numpy as np
import onnx
from onnx import TensorProto, helper, numpy_helper

rng = np.random.default_rng(42)
b = (rng.standard_normal((4096, 1024)) * 0.01).astype(np.float16)
c = np.zeros((4096,), dtype=np.float16)

model = helper.make_model(
    helper.make_graph(
        [helper.make_node("Gemm", ["A", "B", "C"], ["Y"], transB=1)],
        "fp16-gemm-repro",
        [helper.make_tensor_value_info("A", TensorProto.FLOAT16, [14, 1024])],
        [helper.make_tensor_value_info("Y", TensorProto.FLOAT16, [14, 4096])],
        [numpy_helper.from_array(b, "B"), numpy_helper.from_array(c, "C")],
    ),
    opset_imports=[helper.make_opsetid("", 20)],
)
model.ir_version = 10
onnx.checker.check_model(model)
onnx.save(model, "fp16_gemm.onnx")

2. Reference either version in a net10.0 console project:

<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.29.0" />

3. Run this benchmark:

using System.Diagnostics;
using Microsoft.ML.OnnxRuntime;

using var options = new SessionOptions
{
    GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_ALL,
    IntraOpNumThreads = 16,
    InterOpNumThreads = 1,
    ExecutionMode = ExecutionMode.ORT_SEQUENTIAL,
};
using var session = new InferenceSession("fp16_gemm.onnx", options);
using var runOptions = new RunOptions();

var input = new Float16[14 * 1024];
using var value = OrtValue.CreateTensorValueFromMemory(input, [14L, 1024L]);
var inputs = new Dictionary<string, OrtValue> { ["A"] = value };

for (int i = 0; i < 3; i++)
    using (session.Run(runOptions, inputs, ["Y"])) { }

var sw = Stopwatch.StartNew();
const int iterations = 20;
for (int i = 0; i < iterations; i++)
    using (session.Run(runOptions, inputs, ["Y"])) { }
sw.Stop();

Console.WriteLine($"ORT={OrtEnv.Instance().GetVersionString()}");
Console.WriteLine($"mean_ms={sw.Elapsed.TotalMilliseconds / iterations:F3}");

4. Run once with 1.28.0 and once with 1.29.0.

Observed:

ORT=1.28.0
mean_ms=0.121

ORT=1.29.0
mean_ms=430.339

Urgency

No response

Platform

Windows

OS Version

Windows 11, build 26200

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.29.0; regression baseline 1.28.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

Yes

[hariharans29]

Thanks ! We will take a look !

[Colm-in-Arm]

Hello WithSalt.

This is an ugly problem.

As you say the fallback from the unsupported FP16 operation is going back to Eigen rather than the cast to FP32 and SGEMM path used in 1.28. This does appear to make a substantial performance difference.

There are a couple of approaches we can take. The quick fix is to catch the FP16 operations not supported by the new MLAS path in insert_cast_transformer.cc and avoid the problem. The better fix would be to support Gemm(transB=1) and the other cases where SGEMM had previously worked.

You didn't specify an urgency above. Is this a critical problem for you?

Colm.

[hariharans29]

Will sync offline with Colm Donelan (@Colm-in-Arm) and try and get it to the 1.28 state for fp16 (atleast for x64).

[tianleiwu]

Hariharan Seshadri (@hariharans29), I started a Copilot session to prototype. We can guide the copilot if needed. I think there are two solutions:
(1) we detect whether fp16 is supported natively. If not, we fallback to fp32 (Need temp buffer for fp32 inputs and ouputs).
Some improvement can be done here (like using tiling) to avoid triple memory usage.
(2) we do not register fp16 gemm if machine does not support it natively.

[hariharans29]

Tianlei Wu (@tianleiwu) : we detect whether fp16 is supported natively -> How to distinguish betwen Eigen supported fp16 and MLAS supported HalfGemm and how to define "native" ?

[hariharans29]

Btw- the copilot solution looks good to me. We only register the fp16 kernels when MLAS_F16VEC_INTRINSICS_SUPPORTED or MLAS_TARGET_RISCV64 is defined (i.e.) ties it to MLAS support only which works for me. It will allow the ARM team to build on their fp16 PRs on their cadence and also preserve x64 behavior.

[withsalt]

Hello WithSalt.

This is an ugly problem.

As you say the fallback from the unsupported FP16 operation is going back to Eigen rather than the cast to FP32 and SGEMM path used in 1.28. This does appear to make a substantial performance difference.

There are a couple of approaches we can take. The quick fix is to catch the FP16 operations not supported by the new MLAS path in insert_cast_transformer.cc and avoid the problem. The better fix would be to support Gemm(transB=1) and the other cases where SGEMM had previously worked.

You didn't specify an urgency above. Is this a critical problem for you?

Colm.

Thanks!
No rush, I can still use 1.28.