MonoAOT: LLVM 23 regressions in FullAOT JIT test outcomes

[pavelsavara]

Summary

Observed on PR branch emsdk-upgrade / #129299: runtime-llvm linux-x64 FullAOT runtime tests regress under LLVM 23 (e.g. nullabletypes returns Expected 100, Actual 666). Each affected test is disabled via [ActiveIssue("https://github.com/dotnet/runtime/issues/129508", ...)] on the emsdk-upgrade branch. The failures have multiple distinct root causes (see below).

All scenarios now have fixes in flight.

Failing scenarios

• nullabletypes — JIT/Directed/nullabletypes/castclassvaluetype, JIT/Directed/nullabletypes/Desktop/boxunboxvaluetype (Expected 100, Actual 666) — fixed by [mono][amd64] Pass small straddling vtypes in registers in both backends #129702
• Runtime_105619 — JIT/Regression/JitBlue/Runtime_105619 — collectible-ALC load + reflection invoke (not codegen) — covered by [mono] Fix re-entrant AssemblyLoadContext resolution stack overflow under full-AOT #129693
• call05_large — JIT/jit64/localloc/call/call05_large (localloc + jmp) — fixed by [mono] AOT-compile methods that use the IL jmp opcode #129708
• call05_small — JIT/jit64/localloc/call/call05_small (localloc + jmp) — fixed by [mono] AOT-compile methods that use the IL jmp opcode #129708
• WPF_3226 — JIT/Regression/JitBlue/WPF_3226 (Marshal.StructureToPtr / Marshal.PtrToStructure marshalling) — fixed by [mono] AOT-compile marshalling wrappers for layout classes with class-typed fields #129710
• b143840 — JIT/Regression/CLR-x86-JIT/V1.1-M1-Beta1/b143840 (synchronized methods / exception throw) — fixed by [mono] Keep in-flight exception alive across LLVM resume unwind #129713
• UnitTest_GVM_TypeLoadException — Loader/classloader/MethodImpl/CovariantReturns/UnitTest (GVM covariant returns + TypeLoadException) — fixed by [mono] Defer NEWOBJ type-load failures to runtime in AOT mode #129715

Fixes

• nullabletypes → [mono][amd64] Pass small straddling vtypes in registers in both backends #129702 — [mono][amd64] Pass small straddling vtypes in registers in the LLVM backend.
  Root cause: a ≤16-byte all-integer value type whose nested field straddles the 8-byte SysV eightbyte boundary (e.g. Nullable<T> with an 8-byte T) was passed byval by the partially-shared caller — which LLVM 18+ lowers onto the stack — while the concrete callee reads it from two integer registers, corrupting the value. Fixed by passing such vtypes in registers in the amd64 LLVM call-info. amd64-only.

• Runtime_105619 → [mono] Fix re-entrant AssemblyLoadContext resolution stack overflow under full-AOT #129693 — [mono] Fix re-entrant AssemblyLoadContext resolution stack overflow under full-AOT.
  This test is not a codegen failure: its TestEntryPoint loads the assembly into a collectible AssemblyLoadContext and invokes a method via reflection (the Fuzzlyn payload itself NREs on uninitialized statics and is swallowed by try {} catch {}), so the only failure mode is a process crash during the ALC load/invoke — the re-entrant ALC resolution stack overflow that [mono] Fix re-entrant AssemblyLoadContext resolution stack overflow under full-AOT #129693 fixes. (Structural match; merged-harness confirmation pending.)

• call05_large / call05_small → [mono] AOT-compile methods that use the IL jmp opcode #129708 — [mono] AOT-compile methods that use the IL jmp opcode.
  Root cause: these tests jmp to another method from a method that also uses localloc. The non-llvm_only CEE_JMP path emitted an OP_TAILCALL and called DISABLE_AOT(), so the method was excluded from the AOT image; under full-AOT (aot-only) the call then tried to JIT it and failed. Fixed by emitting jmp as a plain call + return for AOT, keeping the method in the image. Pre-existing; newly exposed by the x64 Mono full-AOT leg.

• WPF_3226 → [mono] AOT-compile marshalling wrappers for layout classes with class-typed fields #129710 — [mono] AOT-compile marshalling wrappers for layout classes with class-typed fields.
  Root cause: can_marshal_struct() in the AOT compiler did not handle MONO_TYPE_CLASS fields, so a [StructLayout] class whose fields are themselves marshalable layout classes (e.g. MINMAXINFO with two POINT fields) was treated as non-marshalable and its StructureToPtr/PtrToStructure wrappers were not emitted into the AOT image; the call then failed to JIT under aot-only. Fixed by recursing into can_marshal_struct() for class-typed fields. Pre-existing; newly exposed by the x64 Mono full-AOT leg.

• b143840 → [mono] Keep in-flight exception alive across LLVM resume unwind #129713 — [mono] Keep in-flight exception alive across LLVM resume unwind.
  Root cause: when unwinding through an LLVM-compiled finally/fault handler, the in-flight exception object was stored as a raw pointer in ResumeState.ex_obj (marked /* FIXME: GC */) across the managed handler (e.g. the Monitor.Exit of a synchronized wrapper). That handler can reach a GC safepoint, and a moving GC relocates the exception, leaving the pointer stale; mono_resume_unwind then passes it to mono_object_isinst_checked during the catch search and dereferences a garbage MonoClass → intermittent SIGSEGV under load. Fixed by keeping the exception alive in a pinned GC handle across the resume window (as the llvmonly path already does). Pre-existing; newly exposed by the x64 Mono full-AOT leg.

• UnitTest_GVM_TypeLoadException → [mono] Defer NEWOBJ type-load failures to runtime in AOT mode #129715 — [mono] Defer NEWOBJ type-load failures to runtime in AOT mode.
  Root cause: each of the four RunInvalidTestN helpers does newobj of a type with an invalid covariant-return override. Resolving the constructor fails with a type-load error during AOT compilation, so the AOT compiler aborted and excluded the whole method from the image; under full-AOT (aot-only) the call then attempted a JIT compile and threw ExecutionEngineException instead of the expected TypeLoadException, which the test's catch did not handle. Fixed by deferring the type-load failure to a runtime throw in AOT (turning the method into one that throws TypeLoadException, as the ldfld path already does), matching the JIT behavior. The IsNativeAot annotation is kept (NativeAOT has the same limitation). Pre-existing; newly exposed by the x64 Mono full-AOT leg.

Note

This issue description was updated with the assistance of GitHub Copilot.

[pavelsavara]

Helix Failure Evidence — Build 1464283

Job: linux-x64 Release AllSubsets_Mono_LLVMFULLAOT_RuntimeTests llvmfullaot
AzDO log (Send tests to Helix): https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/1464283/logs/110
Helix job: https://helix.dot.net/api/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems?api-version=2019-06-17

Total: 11 errors across 4 work items.

---

Work item: jit64_5

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/jit64_5/console

Failure 1 — call05_large — System.ExecutionEngineException attempting to JIT in full-AOT mode:
16:25:54.447 Running test: JIT/jit64/localloc/call/call05_large/call05_large.dll System.ExecutionEngineException: Attempting to JIT compile method 'int LocallocTest:func0 (int,int,int,int,int,int,int,int,int*,int*)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. at LocallocTest.Main() at Program.<<Main>$>g__TestExecutor4|0_5(...) 16:25:54.450 Failed test: JIT/jit64/localloc/call/call05_large/call05_large.dll

Failure 2 — call05_small — same ExecutionEngineException:
16:25:54.451 Running test: JIT/jit64/localloc/call/call05_small/call05_small.dll System.ExecutionEngineException: Attempting to JIT compile method 'int LocallocTest:func0 (int,int,int,int,int,int,int,int,int*,int*)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. at LocallocTest.Main() at Program.<<Main>$>g__TestExecutor5|0_6(...) 16:25:54.453 Failed test: JIT/jit64/localloc/call/call05_small/call05_small.dll

Root cause: jmp IL opcode in unc0 forces a non-inlinable tail call that cannot be AOT-compiled by Mono LLVM FullAOT. The JIT is invoked at runtime, which is forbidden in --full-aot mode.

---

Work item: Directed_1

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Directed_1/console

Failures 3–10: all 8 variants of �oxunboxvaluetype and castclassvaluetype nullable tests fail with wrong return code. Reproduced error pattern (shown for castclassvaluetype_r and castclassvaluetype_ro):

--- NotEmptyStructConstrainedGenQ<int>? s = Helper.Create(default(NotEmptyStructConstrainedGenQ<int>)) --- Test FAILED System.Exception: [False]!=[True]: BoxUnboxToNQGenC at Assert.AreEqual(Boolean result, Boolean expected, String comment) at NullableTest30.Run() at Test_castclassvaluetype.TestEntryPoint() Xunit.Sdk.EqualException: Assert.Equal() Failure: Values differ Expected: 100 Actual: 666 16:26:05.066 Failed test: JIT/Directed/nullabletypes/castclassvaluetype_r/castclassvaluetype_r.dll

The Expected: 100 / Actual: 666 pattern confirms TestEntryPoint catches an exception and returns error code 666 instead of the success code 100. This affects all 4 castclassvaluetype variants (_d, _do, _r, _ro) and all 3 boxunboxvaluetype variants (_do, _r, _ro).

Failing tests in this work item:

• JIT/Directed/nullabletypes/Desktop/boxunboxvaluetype_do
• JIT/Directed/nullabletypes/Desktop/boxunboxvaluetype_r
• JIT/Directed/nullabletypes/Desktop/boxunboxvaluetype_ro
• JIT/Directed/nullabletypes/castclassvaluetype_d
• JIT/Directed/nullabletypes/castclassvaluetype_do
• JIT/Directed/nullabletypes/castclassvaluetype_r
• JIT/Directed/nullabletypes/castclassvaluetype_ro

---

Work item: Regression_o_3

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Regression_o_3/console

Failure 11 — WPF_3226 — System.ExecutionEngineException trying to JIT a Marshal.StructureToPtr wrapper in full-AOT mode:
16:25:55.134 Running test: global::WPF_3226.Test_WPF_3226.TestEntryPoint() System.ExecutionEngineException: Attempting to JIT compile method '(wrapper other) void WPF_3226.MINMAXINFO:StructureToPtr (object,intptr,bool)' while running in aot-only mode. See https://learn.microsoft.com/xamarin/ios/internals/limitations for more information. at System.Runtime.InteropServices.Marshal.StructureToPtr[MINMAXINFO](MINMAXINFO structure, IntPtr ptr, Boolean fDeleteOld) at WPF_3226.Test_WPF_3226.WmGetMinMaxInfo(IntPtr lParam) at WPF_3226.Test_WPF_3226.TestEntryPoint() 16:25:55.137 Failed test: global::WPF_3226.Test_WPF_3226.TestEntryPoint()

Root cause: Marshal.StructureToPtr emits a wrapper method at runtime. Mono LLVM FullAOT cannot JIT this wrapper. This failure is a structural limitation of using generic Marshal.StructureToPtr in FullAOT mode, unrelated to the LLVM version bump.

---

Work item: Loader

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Loader/console

Failure — UnitTest_GVM_TypeLoadException — reported as failed by the Helix task but the work item log shows

Helix Failure Evidence - Build 1464283

Job: linux-x64 Release AllSubsets_Mono_LLVMFULLAOT_RuntimeTests llvmfullaot
AzDO log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/1464283/logs/110
Helix job: https://helix.dot.net/api/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems?api-version=2019-06-17

Total: 11 errors across 4 work items.

---

Work item: jit64_5

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/jit64_5/console

Failure 1 - call05_large - System.ExecutionEngineException attempting to JIT in full-AOT mode:

16:25:54.447 Running test: JIT/jit64/localloc/call/call05_large/call05_large.dll
System.ExecutionEngineException: Attempting to JIT compile method 'int LocallocTest:func0 (int,int,int,int,int,int,int,int,int*,int*)' while running in aot-only mode.
   at LocallocTest.Main()
16:25:54.450 Failed test: JIT/jit64/localloc/call/call05_large/call05_large.dll

Failure 2 - call05_small - same ExecutionEngineException:

16:25:54.451 Running test: JIT/jit64/localloc/call/call05_small/call05_small.dll
System.ExecutionEngineException: Attempting to JIT compile method 'int LocallocTest:func0 (int,int,int,int,int,int,int,int,int*,int*)' while running in aot-only mode.
   at LocallocTest.Main()
16:25:54.453 Failed test: JIT/jit64/localloc/call/call05_small/call05_small.dll

Root cause: jmp IL opcode in func0 forces a non-inlinable tail call that cannot be AOT-compiled. The JIT is invoked at runtime, which is forbidden in --full-aot mode.

---

Work item: Directed_1

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Directed_1/console

Failures 3-10: all 8 variants of boxunboxvaluetype and castclassvaluetype nullable tests. Pattern (castclassvaluetype_r shown):

--- NotEmptyStructConstrainedGenQ<int>? s = Helper.Create(default(NotEmptyStructConstrainedGenQ<int>)) ---
Test FAILED
System.Exception: [False]!=[True]: BoxUnboxToNQGenC
   at NullableTest30.Run()
   at Test_castclassvaluetype.TestEntryPoint()
Xunit.Sdk.EqualException: Assert.Equal() Failure: Values differ
Expected: 100
Actual:   666
16:26:05.066 Failed test: JIT/Directed/nullabletypes/castclassvaluetype_r/castclassvaluetype_r.dll

The Expected 100 / Actual 666 confirms TestEntryPoint catches an exception and returns error code 666. NullableTest30.BoxUnboxToNQGenC returns False when True is expected. Affects all 4 castclassvaluetype variants and all 3 boxunboxvaluetype variants.

---

Work item: Regression_o_3

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Regression_o_3/console

Failure 11 - WPF_3226 - ExecutionEngineException on Marshal.StructureToPtr wrapper:

16:25:55.134 Running test: global::WPF_3226.Test_WPF_3226.TestEntryPoint()
System.ExecutionEngineException: Attempting to JIT compile method '(wrapper other) void WPF_3226.MINMAXINFO:StructureToPtr (object,intptr,bool)' while running in aot-only mode.
   at System.Runtime.InteropServices.Marshal.StructureToPtr[MINMAXINFO](MINMAXINFO structure, IntPtr ptr, Boolean fDeleteOld)
   at WPF_3226.Test_WPF_3226.WmGetMinMaxInfo(IntPtr lParam)
   at WPF_3226.Test_WPF_3226.TestEntryPoint()
16:25:55.137 Failed test: global::WPF_3226.Test_WPF_3226.TestEntryPoint()

Root cause: Marshal.StructureToPtr emits a wrapper method at runtime. Mono LLVM FullAOT cannot JIT this wrapper. This is a structural AOT limitation unrelated to the LLVM version bump.

---

Work item: Loader

Console log: https://helix.dot.net/api/2019-06-17/jobs/f3bc3431-e44d-47b3-b4ce-492dd151bba7/workitems/Loader/console

Failure: UnitTest_GVM_TypeLoadException - reported failed by Helix task but work item log ends with 0 failures reported, exit code 0. The log is very large (~384KB) - the failure detail may be in an earlier test run stripe or in the uploaded test results XML rather than the console log.

---

Summary table

Work item	Test	Root cause
jit64_5	call05_large, call05_small	jmp IL causes JIT attempt in full-AOT
Directed_1	7x nullable boxunbox/castclass	Wrong logic result - NullableTest30 BoxUnboxToNQGenC
Regression_o_3	WPF_3226	Marshal.StructureToPtr wrapper not AOT-compiled
Loader	UnitTest_GVM_TypeLoadException	Inconclusive from console log