perf(EPv2): size the gfx1250 combine pull tile against the LDS budget - #608
Conversation
The TDM pull path took its tile from a fixed chunk count, so the LDS it needed was warpNum * srcMax * (hidden / MORI_COMB_TDM) * sizeof, where srcMax is how many distinct ranks a token landed on. At world 4 srcMax is 4 and that product is 230 KB, inside the 320 KB budget. At world 8 with topk 6 it is 6, the product is 345 KB, _cPullOk goes false, and combine drops to the per-element fallback that reads peer memory with the fabric latency exposed. Measured on two gfx1250 nodes, 8 ranks over scale-up, against this commit's parent, ct=512, hidden 7168, comb 64x8, ITERS=50, median of 3 alternating rounds, CHECK on: world 8 topk 6 combine 619.2 us / 53.0 GB/s -> 81.1 us / 403.5 GB/s world 8 topk 8 combine 89.8 us / 423.7 GB/s -> 89.5 us / 422.0 GB/s world 4 topk 6 combine 60.9 us / 395.4 GB/s -> 60.7 us / 396.4 GB/s world 4 topk 8 combine 63.2 us / 414.9 GB/s -> 63.1 us / 416.4 GB/s topk 8 at world 8 was never affected: srcMax == npes makes the QUAD gate true and that path sizes its tiles separately, so it never reads this budget. The broken band was topk 6, 7 and topk >= 9, with 5 and 8 fine on either side of it, which is why it went unnoticed. Solving for the tile leaves every shape that already fit bit-identical -- world 4 still resolves to 2 chunks and the same 230144 bytes, hence the unchanged rows above -- and takes world size out of the inequality, because srcMax saturates at topk once npes >= topk. Bounding srcMax by min(npes, topk) instead of topk also stops over-reserving when topk > npes. World 16/32/64 are covered by a host-side replica of the expression only (3 chunks, 312320 bytes at topk 8); no machine above 8 ranks was available. Two adjacent changes were measured and left out, each 3/3 rounds slower at world 8 topk 6: letting the QUAD gate take topk < npes (+0.4 us, since a group then has more warps than the token has sources), and dropping the npes <= 4 guard on this path's compaction (+1.4 us, the ballot costs more than the reduce iterations it saves, which _CROW_DEAD already skipped).
Two ternaries in the previous commit were wrapped by hand where the pinned clang-format (mirrors-clang-format v20.1.8, --style=file) breaks them differently. Whitespace only, and no token is added or removed, so the measurements in the previous commit still describe this code.
|
Verified the diagnosis independently and it reproduces exactly. Recomputing the LDS need under That is the odd band the description calls out, and the reason topk 8 escapes is exactly that The three numbers in the description reproduce exactly too: world 4 topk 6 → 2 chunks / 230144 (unchanged), world 8 topk 6 → 3 chunks / 234240 (now fits), world 16 topk 8 → 3 chunks / 312320. Two things I checked because the new expression introduces a division the old one did not have:
Three notes, none of them correctness: 1. The 2. The world-4 rows are a no-regression control, not a test of the changed path. At world 4 with topk 6 or 8, 3. LGTM. The fix takes world size out of the inequality, leaves every shape that already fit bit-identical, and the description is unusually complete: measurements, why the band looked arbitrary, the two adjacent changes tried and rejected, and an honest note that above 8 ranks the coverage is arithmetic rather than measured. |
…fx1250 The notify loop read the destination's signal slot to confirm the previous recv-count had been consumed, and it did so after the grid barrier had already been waited out. That read goes to uncached peer memory, so it pays a full fabric round trip on every launch even though the slot has long been zero -- fully exposed on the critical path. Its address depends only on destPe, so it needs nothing the barrier provides. Issuing it before the barrier spin lets the round trip overlap the wait. This is the same reordering v1's 1250x body already carries, and the comment there records the same magnitude of win. The wire format is byte-for-byte unchanged: both are pure spin-waits that write nothing, and the signal store still happens after both. No EpArgs or arena change, so no rebuild is required to pick it up. Isolated effect of this commit alone, gfx1250, EP4, hidden 7168, topk 6, bf16, graph mode, median of 3 -- measured before the rebase onto ROCm#608, so against the older base rather than this commit's current parent: tokens dispatch before after delta 512 44.2 us 41.1 us -7.0% 4096 139.2 us 136.1 us -2.2% The follow-up commit removes the barrier spin this one leans on, and carries the definitive numbers for the pair measured against the current parent.
…fx1250 The notify loop read the destination's signal slot to confirm the previous recv-count had been consumed, and it did so after the grid barrier had already been waited out. That read goes to uncached peer memory, so it pays a full fabric round trip on every launch even though the slot has long been zero -- fully exposed on the critical path. Its address depends only on destPe, so it needs nothing the barrier provides. Issuing it before the barrier spin lets the round trip overlap the wait. This is the same reordering v1's 1250x body already carries, and the comment there records the same magnitude of win. The wire format is byte-for-byte unchanged: both are pure spin-waits that write nothing, and the signal store still happens after both. No EpArgs or arena change, so no rebuild is required to pick it up. Isolated effect of this commit alone, gfx1250, EP4, hidden 7168, topk 6, bf16, graph mode, median of 3 -- measured before the rebase onto ROCm#608, so against the older base rather than this commit's current parent: tokens dispatch before after delta 512 44.2 us 41.1 us -7.0% 4096 139.2 us 136.1 us -2.2% The follow-up commit removes the barrier spin this one leans on, and carries the definitive numbers for the pair measured against the current parent.
2 participants
The TDM pull path took its tile from a fixed chunk count, so the LDS it needed was warpNum * srcMax * (hidden / MORI_COMB_TDM) * sizeof, where srcMax is how many distinct ranks a token landed on. At world 4 srcMax is 4 and that product is 230 KB, inside the 320 KB budget. At world 8 with topk 6 it is 6, the product is 345 KB, _cPullOk goes false, and combine drops to the per-element fallback that reads peer memory with the fabric latency exposed.
Measured on two gfx1250 nodes, 8 ranks over scale-up, against this commit's parent, ct=512, hidden 7168, comb 64x8, ITERS=50, median of 3 alternating rounds, CHECK on:
world 8 topk 6 combine 619.2 us / 53.0 GB/s -> 81.1 us / 403.5 GB/s
world 8 topk 8 combine 89.8 us / 423.7 GB/s -> 89.5 us / 422.0 GB/s
world 4 topk 6 combine 60.9 us / 395.4 GB/s -> 60.7 us / 396.4 GB/s
world 4 topk 8 combine 63.2 us / 414.9 GB/s -> 63.1 us / 416.4 GB/s
topk 8 at world 8 was never affected: srcMax == npes makes the QUAD gate true and that path sizes its tiles separately, so it never reads this budget. The broken band was topk 6, 7 and topk >= 9, with 5 and 8 fine on either side of it, which is why it went unnoticed.
Solving for the tile leaves every shape that already fit bit-identical -- world 4 still resolves to 2 chunks and the same 230144 bytes, hence the unchanged rows above -- and takes world size out of the inequality, because srcMax saturates at topk once npes >= topk. Bounding srcMax by min(npes, topk) instead of topk also stops over-reserving when topk > npes.
World 16/32/64 are covered by a host-side replica of the expression only (3 chunks, 312320 bytes at topk 8); no machine above 8 ranks was available.
Two adjacent changes were measured and left out, each 3/3 rounds slower at world 8 topk 6: letting the QUAD gate take topk < npes (+0.4 us, since a group then has more warps than the token has sources), and dropping the npes <= 4 guard on this path's compaction (+1.4 us, the ballot costs more than the reduce iterations it saves, which _CROW_DEAD already skipped).