Skip to content

[TEST ONLY][DO NOT MERGE] Bump LLVM pin to 739eaa08 to fix gfx1250 entry prologue - #1088

Open
Phil-amd wants to merge 1 commit into
phil/bump-llvm-941a04e6from
phil/test-bump-llvm-739eaa08
Open

[TEST ONLY][DO NOT MERGE] Bump LLVM pin to 739eaa08 to fix gfx1250 entry prologue#1088
Phil-amd wants to merge 1 commit into
phil/bump-llvm-941a04e6from
phil/test-bump-llvm-739eaa08

Conversation

@Phil-amd

@Phil-amd Phil-amd commented Sep 2, 2026

Copy link
Copy Markdown
Member

Test only — do not merge. Opened to validate a fix on gfx1250 hardware.
Stacked on #1051 (phil/bump-llvm-941a04e6), so the diff here is the one-line
pin change. If this is confirmed, the right outcome is to move #1051's own pin
and close this.

Problem

#1051 is reported to abort on gfx1250 for any kernel, e.g.

python3 tests/kernels/test_gemm_fp8fp4_gfx1250.py -mode mxscale_a8w4 \
    -mnk 1024,1024,1024 -tiles 128,128,128 -warps 2,2 -nb 4
HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION: The agent attempted to access
memory beyond the largest legal address.

with the faulting PC on the global_prefetch_b8 at the top of the kernel.

Root cause

The global_prefetch_b8 is not emitted by FlyDSL. The only global_prefetch
emitter in the tree is l2_prefetch_tile
(python/flydsl/expr/rocdl/tdm_ops.py), and it has no callers. The
instruction comes from SIInsertWaitcnts.cpp, which prepends a hardware
entrypoint prologue to every entry function whose subtarget has
requires-initial-unclause-vmem (gfx1250/gfx1251). That is why any kernel
reproduces it, not just this GEMM.

Upstream reworked that prologue three times in three weeks, and
941a04e69ee8 (2026-08-09) sits in the middle of the sequence:

upstream commit date prologue ancestor of 941a04e6?
ae98c7d9d386 (#214127) 08-05 global_prefetch_b8 v0, s[0:1]v0, null yes
1109d68feb1b (#215450) 08-11 back to a real pair: s_mov_b64 s[64:65], 0 + v_nop + v0, s[64:65] no
739eaa08d45e (#216897) 08-18 same, plus the REPLAY_MODE s_setreg hoisted ahead of it no

So #1051 picks up the null-saddr form together with neither of the two
commits that walk it back. Upstream reverting a just-merged change twice is
itself the signal that the null form is wrong on hardware, and it matches the
symptom: with no legal zero base in saddr, the prologue prefetch addresses
outside the aperture.

Note #1051's stated motivation is #214885
("Keep TDM counter low"); the commit that bites is a different one in the same
range.

This change

Moves the pin to 739eaa08d45e, which includes both follow-ups.

Validation asked for

Needs a gfx1250 machine — I have no such hardware locally, and everything above
is static analysis of the upstream history, not a reproduction.

  1. Confirm the built prologue now contains s_mov_b64 s[64:65], 0:
    llvm-objdump -d --mcpu=gfx1250 <kernel>.hsaco | head -20
  2. Re-run the reported command.

One caveat worth checking while there: the reported disassembly prints the saddr
as off, not the null that 941a04e69ee8 should emit, so the LLVM deployed on
that machine may not be exactly at the pin. Worth confirming, since it changes
whether the bad window is the whole story.

Disable the JIT disk cache on both sides when comparing
(FLYDSL_RUNTIME_ENABLE_CACHE=0) — the cache key does not include the LLVM
version, so a stale HSACO can otherwise mask the difference in either direction.

…try prologue

Test-only bump on top of #1051 to move the pin from 941a04e69ee8 (2026-08-09)
to 739eaa08d45e (2026-08-18). Not for merge; opened to validate on gfx1250.

#1051 pins 941a04e69ee8, which lands in the middle of an upstream sequence that
reworks the gfx1250/gfx1251 hardware entrypoint prologue three times. That
prologue is inserted by SIInsertWaitcnts for every entry function whenever the
subtarget has requires-initial-unclause-vmem, so it affects every kernel:

  ae98c7d9d386 (#214127, 08-05)  global_prefetch_b8 v0, s[0:1] -> v0, null
  1109d68feb1b (#215450, 08-11)  reverts to a real register pair:
                                   s_mov_b64 s[64:65], 0
                                   v_nop
                                   global_prefetch_b8 v0, s[64:65]
  739eaa08d45e (#216897, 08-18)  same sequence, and hoists the REPLAY_MODE
                                   s_setreg ahead of it

Only the first is an ancestor of 941a04e69ee8, so #1051 picks up the null-saddr
form without either of the two commits that walk it back. Upstream undoing a
just-merged change twice is the tell that the null form is wrong on hardware,
and it matches the report against #1051 on gfx1250: a plain GEMM aborts with
HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION and the faulting PC sits on the
prologue global_prefetch_b8. With no legal zero base in saddr the prefetch
addresses outside the aperture.

The instruction is not FlyDSL's. The only global_prefetch emitter in the tree is
l2_prefetch_tile in python/flydsl/expr/rocdl/tdm_ops.py, which has no callers,
which is why any kernel reproduces it.

Moving to 739eaa08d45e takes both follow-ups. gpu::LaunchFuncOp is untouched in
941a04e69ee8..739eaa08d45e, so the FlyToROCDL adaptation from #1051 carries over
unchanged, and FlyDSL implements no CallOpInterface op, so the interface
verification tightened by #214724 in the same range does not apply.

Unverified: the reported disassembly prints the saddr as `off` rather than the
`null` the pin should emit, so the deployed LLVM may not be exactly 941a04e69ee8.
Confirm the built prologue contains s_mov_b64 s[64:65], 0 on the test machine.
@Phil-amd
Phil-amd marked this pull request as ready for review September 2, 2026 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant