[mono][amd64] Pass small straddling vtypes in registers in the LLVM backend

[pavelsavara]

Summary

On amd64, a Γëñ16-byte all-integer value type whose nested field crosses the 8-byte SysV eightbyte boundary is passed on the stack as a byval struct by the Mono LLVM backend, while a fully-flattened instantiation of the same layout is passed in two integer registers. These two classifications must agree because partially-shared generic code calls the concrete method, and recent LLVM lowers small byval structs onto the stack instead of into registers ΓÇö so the value is corrupted across the call. In practice this shows up only under Mono LLVM full-AOT on x64, e.g. boxing/unboxing a Nullable<struct-with-a-nullable-field> in shared generic code returns the wrong value.

Minimal repro

using System;
public struct GenQ<T> where T : struct { public T? Field; }   // 12 bytes: { bool hasValue, int value }
public static class Repro {
    static bool Gen<T>(T? o) where T : struct {                // partially-shared generic
        var v = (GenQ<int>)(ValueType)(object)o;               // box T?, then unbox
        return !v.Field.HasValue;
    }
    static int Main() => Gen<GenQ<int>>(default(GenQ<int>)) ? 100 : 666;
}

• Mono JIT: returns 100.
• Mono LLVM full-AOT on x64 (LLVM 18+): returns 666 ΓÇö the boxed/unboxed value is corrupted.

Root cause

add_valuetype (mini-amd64.c) forces a Γëñ16-byte value type onto the stack (ArgOnStack ΓåÆ LLVMArgVtypeByVal) when a nested field straddles the 8-byte eightbyte boundary, because the JIT cannot place a straddling field in a register pair. For Nullable<GenQ<int>> the value field is at offset 4, size 8 (it crosses byte 8).

The concrete, fully-flattened instantiation ({ bool@0, bool@4, int@8 }) has no straddling field and is passed in two integer registers (ArgValuetypeInReg).

Partially-shared generic code calls the concrete method, so the conventions must match: the partially-shared caller sees the value type as an opaque type parameter (one straddling field ΓåÆ byval) while the concrete Nullable<T>:Box callee sees the flattened layout (ΓåÆ registers). Older LLVM passed the small byval struct in registers so they matched; LLVM 18+ lowers byval onto the stack, so the caller writes the value to the stack while the callee reads it from registers ΓåÆ corruption.

Fix

Mark all-integer Γëñ16-byte straddling vtypes (ArgInfo.llvm_inreg_straddle) and have get_llvm_call_info pass them in two integer registers (LLVMArgVtypeInReg), matching the flattened classification and restoring the pre-LLVM-18 behavior. The non-LLVM JIT path is unchanged; float-containing structs keep the byval path.

Scope x64 only

• arm64 classifies by size (no straddle rule) and uses a register-array form (LLVMArgAsIArgs), so caller and callee already agree.
• x86 / arm / riscv have no straddle rule.
• wasm / llvmonly use a different, already-correct nullable-box path.

Validation

Built Mono + LLVM full-AOT on x64 and ran the JIT/Directed/nullabletypes test sources (castclassvaluetype, which covers every nullable type ΓÇö int?ΓǪGuid?, the generic-struct cases, and float?/double?/decimal?): all pass under both full-AOT and JIT with the fix; without it the generic-struct nullable cases fail. The minimal repro above goes from 666 ΓåÆ 100.

Alternatives considered

1. Patch the Mono LLVM fork to pass Γëñ16-byte INTEGER byval in registers (restore the old lowering). Uniform, but lives in dotnet/llvm-project and re-introduces reliance on non-standard byval register passing.
2. Route the shared nullable-box through a gsharedvt-out wrapper. Works for the fully-gsharedvt variant, but the hot path here is partially-shared (cfg->gsharedvt == 0), whose call passes the value byval (not by-ref), so the out-wrapper expects the wrong convention and faults.
3. This change ΓÇö fix the classification at its source on the only affected arch; smallest, leaves the JIT untouched, and makes both sides agree regardless of LLVM's byval lowering.

Note

This pull request was prepared with the assistance of GitHub Copilot.

---

Part of the MonoAOT LLVM 23 full-AOT regression set tracked by #129508. Built and tested together with the Emscripten 5.0.6 / LLVM 23 bump on #129396, where the re-enabled nullabletypes tests (boxunboxvaluetype / castclassvaluetype) pass on the runtime-llvm AllSubsets_Mono_LLVMFULLAOT_RuntimeTests leg.

Contributes to #129508.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @vitek-karas
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR adjusts Mono’s amd64 (SysV) calling-convention classification for the LLVM backend so that certain small (≤16-byte) all-integer value types that were previously forced onto the stack due solely to “straddling field” layout are instead passed in integer registers, aligning the opaque/gshared classification with the fully-flattened classification and avoiding ABI mismatches in LLVM AOT codegen.

Changes:

• Add an ArgInfo.llvm_inreg_straddle flag to mark “stack-only due to straddle” small all-integer structs.
• In add_valuetype(), detect straddling fields and set llvm_inreg_straddle for eligible non-return structs.
• In mono_arch_get_llvm_call_info(), when an argument is ArgOnStack but marked llvm_inreg_straddle, pass it as LLVMArgVtypeInReg (two integer registers) instead of LLVMArgVtypeByVal.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/mono/mono/mini/mini-amd64.h	Adds a new ArgInfo bitflag to annotate the “straddle-induced stack” case for LLVM.
src/mono/mono/mini/mini-amd64.c	Computes the new flag during SysV vtype classification and uses it to select an LLVM in-register lowering for affected arguments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.