sm_120 body NOPs track FP64 issue-rate scheduling at corpus scale

[delicious-pancakes]

Summary

Across cublasLt, cusolver, cusparse on CUDA 13.1, the body-only sm_120 NOP
fraction (10–32%) is generated by ptxas inserting explicit NOPs with
stall-count control codes around FP64 instructions (DFMA / DMUL / DADD).
The compiler emits 4-NOP runs at mean stall=12 cycles per NOP. A direct
dependent chain places the consumer exactly 64 cycles after its producer,
but a controlled eight-chain kernel now shows the same 4-NOP spacing
between register-disjoint DFMAs. The NOP pattern is therefore not caused
solely by dependent-result latency.

The cross-arch shift is evidence of architecture-specific ptxas FP64
issue-rate scheduling, not a post-link finalization artifact and not an
FP32 phenomenon. GB203's 2 FP64 units/SM, versus GH100's 64, are
consistent with the reduced issue capacity; the exact hardware mechanism
and scheduler rationale remain hypotheses.

Result — corpus matrix

Library	sm_80 NOP	sm_120 NOP	body-only sm_80	body-only sm_120
cublasLt	1.35%	24.72%	0.01%	23.82%
cusolver	1.00%	32.11%	0.26%	31.77%
cusparse	1.08%	10.62%	0.26%	9.71%

Body-only counts strip the trailing run of NOP-encoded instructions per
kernel (alignment padding). On sm_80, 74–99% of NOPs are tail alignment;
on sm_120, only 1–10% are tail. The Blackwell shift is body-internal.

Result — NOP-run placement

For each maximal run of body NOPs, count the predecessor opcode
(last non-NOP before) and successor opcode (first non-NOP after) plus
run length:

	k=4 runs	DFMA succ	{DFMA,DMUL,DADD} succ
cublas_sm120	73%	68%	85%
cusolver_sm120	81%	51%	86%
cusparse_sm120	67%	53%	80%

NOP-run length is sharply bimodal at k=4 (67–81%). Successor opcodes
are dominated by the FP64 family; FFMA does not appear in the top
successors. The mechanism is FP64-specific, observed across all three
libraries.

Negative control: cusolver on sm_80 shows 99.96% length-1 runs with
predecessors / successors that are control-flow / sync ops (ISETP, CALL,
IMAD, BSSY) — zero FP64 signature. The 4-NOP × FP64 pattern is unique
to Blackwell consumer SMs.

Result — NOP stall-code distribution

Each 16-byte SASS instruction carries a control word in the upper bits;
its low 4 bits are the stall count (0–15) the warp scheduler waits
before issuing the next instruction. Histogram for every NOP:

	total NOPs	mean stall	stall=15 %	stall=4 %
cublas_sm120	1,251,947	12.17	72.1%	18.0%
cusolver_sm120	3,904,290	12.22	73.6%	18.5%
cusparse_sm120	636,530	11.80	70.4%	17.5%
cusolver sm_80	24,819	6.13	0%	0.01%
(single-mode at stall=6, 97.5%)

The bimodal distribution stall=15 + stall=4 is the signature of active
ptxas scheduling. A post-link / finalizer padding mechanism would emit
either stall=0 or a single uniform fill value. Compare the sm_80
control: a single dominant mode at stall=6, consistent with sync-
barrier insertion.

Cycle reconciliation with arXiv:2507.10789v2

Jarmusch, Graddon, and Chandrasekaran 2025, "Dissecting the NVIDIA
Blackwell Architecture with Microbenchmarks" (arXiv:2507.10789v2),
report for GB203:

• Pure-FP64 true latency: 63.57 cycles (Table III)
• Pure-FP64 completion interval: 11 cycles per warp-instruction
  (Table III)
• Two-instruction dependent configuration: 37.5 cycles (prose
  following Table III)
• FP64 units per SM: 2 (vs GH100's 64; Table I)

The representative direct dependency accounts for the 64-cycle issue
distance without an inferred pipeline tail:

producer DFMA stall=15
+ NOP stalls 15 + 15 + 15 + 4
= 64 cycles from producer issue to consumer issue

The four NOPs contribute 49 cycles; the producer contributes the first
15. The consumer's own stall belongs to the following issue interval and
is not counted here. The producer's write-barrier and the consumer's wait
mask carry the long readiness dependency, so 64 is the chain cost and the
straight-line schedule selected by ptxas, not the full correctness
requirement.

The aggregate NOP histogram does not by itself reconstruct the paper's
37.5-cycle two-instruction configuration. Connecting that result to the
control codes requires a concrete producer-to-consumer sequence including
the producer stall, every intervening instruction, and the matching
write-barrier/wait identity. No yield value is inferred from the stall
count; the fields must be decoded separately.

Controlled discriminator — independent FP64 chains

The latency interpretation needs a dependency control. The first attempted
ILP kernel was not valid for that purpose: all eight accumulators started
with the same value and performed identical updates, so ptxas collapsed
them by common-subexpression elimination to eight DFMAs.

The corrected kernel loads eight distinct accumulator/multiplier pairs and
stores all eight results separately. It emits 64 DFMAs on both targets. With
CUDA 13.1.115:

controlled kernel	DFMAs	NOPs between first/last DFMA
distinct 8-chain, sm_100	64	0
distinct 8-chain, sm_120	64	251

On sm_120, even adjacent register-disjoint operations are separated:

DFMA R26, R26, R32, 1 ;
NOP ;
NOP ;
NOP ;
NOP ;
DFMA R28, R28, R30, 1 ;

The second DFMA neither reads nor writes the first one's register pairs.
This discriminates the mechanisms: four-NOP padding persists when RAW
dependency latency is available to hide with independent FP64 work. The
pattern is therefore best described as FP64 issue-rate scheduling. The
64-cycle producer-inclusive accounting remains valid for the direct
dependent chain, but it is a separate observation rather than the cause of
the corpus-wide NOP density.

The matched FP32 chain is a negative control: sm_100 and sm_120 each emit
24 FFMAs with no NOPs between the first and last FFMA. Tail/alignment NOPs
are excluded from both controlled counts.

Single-cubin instruction-level cross-check

Pick any cublasLt sm_120 cubin with a heavy DFMA section. Steady-state
in the body shows:

/*0c10*/  DFMA R66, R36, -R42, R66 ;
/*0c20*/  NOP ;
/*0c30*/  NOP ;
/*0c40*/  NOP ;
/*0c50*/  NOP ;
/*0c60*/  DFMA R36, R44,  R40, R48 ;
/*0c70*/  NOP ;
/*0c80*/  NOP ;
/*0c90*/  NOP ;
/*0ca0*/  NOP ;
/*0cb0*/  DFMA R48, R38, -R42, R64 ;
...

Preamble shows DFMA → 4 useful (LDG/IMAD) ops + 3 NOPs → DFMA — the
compiler fills latency slots with reorderable work when available, and
falls back to NOPs only when no other work fits. This matches the k=3
secondary mode (16% of runs) in the histogram exactly.

OBS / INF / HYP discipline

• [OBS] Body-only sm_120 NOP rate: cublasLt 23.82%, cusolver 31.77%,
  cusparse 9.71%. sm_80 baseline ≤0.26% across all three.
• [OBS] NOP-run length is bimodal at k=4 (67–81%) on sm_120 across
  all three libraries; sm_80 is 99.96% length-1.
• [OBS] NOP successors are 80–86% FP64 family (DFMA + DMUL + DADD).
  FFMA does not dominate.
• [OBS] NOP stall codes are bimodal at stall=15 (70–73%) and
  stall=4 (17–19%), mean 11.8–12.2 cycles. sm_80 control is single-
  mode at stall=6 (97.5%).
• [OBS] GB203 has 2 FP64 units/SM vs GH100's 64
  (arXiv:2507.10789v2 Table I).
• [OBS] GB203 pure-FP64 true latency is 63.57c in Table III;
  the paper separately reports 37.5c for its two-instruction dependent
  configuration in the accompanying prose.
• [OBS] In the representative direct chain, producer stall=15 plus
  NOP stalls 15 + 15 + 15 + 4 gives exactly 64 cycles from producer
  issue to consumer issue. The producer writes a scoreboard that the
  consumer waits on.
• [OBS] A corrected eight-chain discriminator emits 64 DFMAs for both
  sm_100 and sm_120. sm_100 places 0 NOPs between the first and last DFMA;
  sm_120 places 251, including four-NOP gaps between register-disjoint
  DFMAs. The matched FP32 chain has 0 body NOPs on both targets.
• [INF] The body NOPs carry active scheduling annotations and enforce
  an FP64 issue-rate policy; they are not uniform finalization padding and
  are not explained solely by dependent-result latency.
• [HYP] GB203's reduced FP64 execution capacity causes the exact
  compiler policy observed here. Confirming that hardware-level causal
  explanation requires instruction-level issue/eligible-warp evidence.

Audit confidence (per FINDINGS.md C0–C6 framework)

• C1: corpus-wide observation, verified against the ground-truth
  NOP encoding signature (code & 0xFFF) == 0x918 on sm_120. Grammar-
  decoded NOP count agrees with the low-12-bit ground-truth count
  exactly on all three sm_120 corpora; small 6-instruction false-
  negative on cusparse_sm80 traced to predicated @P? NOP variants
  (≪0.01% miss).
• C2: confirmed by single-variable variation. Holding library and
  toolkit fixed, varying -arch=sm_NN reproduces both the corpus
  shift and the run-length / stall-code distribution shift.
• C3: per-cubin spot check (instruction stream, predecessor /
  successor / NOP count match the corpus pattern).
• C4: cycle-budget cross-check on a representative barrier-linked
  chain: producer stall plus intervening NOP stalls sum to 64 cycles,
  consistent with the independently measured 63.57-cycle true latency.
  Corpus-wide exact-gap reconstruction and the 37.5-cycle regime remain
  open.
• C5: controlled compiler-output discriminator: a CSE-resistant
  eight-chain kernel retains four-NOP gaps between register-disjoint DFMAs
  on sm_120, while sm_100 packs them and the FP32 control has no body NOPs.
  No NCU issue profile is attached, so the hardware-level cause remains
  open; per-instruction issue and eligible-warp counters would test it.
• C6: 3 libraries × 2 archs corpus, plus per-cubin verification on
  one cubin. Adding a second cubin from a different library would
  strengthen.

Falsification check — finalization-padding alternative

A post-link finalizer / capsule-style padding hypothesis predicts:

post-link prediction	actual evidence
Uniform / null stall codes	bimodal stall=15 + stall=4
No correlation with surrounding opcodes	80–86% FP64-family successors
Insertion at ptxas-output → finalizer boundary, not interleaved	NOPs interleaved every 4 instructions inside DFMA chains
sm_80 should also show post-link-shaped padding	sm_80 has none of these features

The stall-code evidence alone is enough to reject post-link padding as
the sole cause. A direct falsifier (rebuild with --binary-kind=sass
to bypass any finalization stage and re-measure NOP density) would
close this independently.

Reproduction packet

The minimal reproducibility packet requested below is now public:

• packet overview and files
• immutable packet revision 9d8059b

It includes the corrected CSE-resistant FP64 discriminator, matched FP32
control, short SASS excerpts and cubin hashes, exact CUDA 13.1.115 commands
and tool hashes, the public analyzer commits, all six aggregate corpus
outputs, and hashes for the six local OR2 corpus stores. No NVIDIA cubin or
CUDA library binary is redistributed.

Methodology / reproducibility

Anyone with the CUDA 13.1 toolkit can reproduce the per-cubin numbers:

# Extract per-arch cubins
cuobjdump --extract-elf "sm_120" \
    /usr/local/cuda-13.1/targets/x86_64-linux/lib/libcublasLt.so

# Total NOP fraction on a single cubin
nvdisasm libcublasLt.*.sm_120.cubin 2>/dev/null \
  | awk '/^[ \t]*\/\*[0-9a-f]+\*\/[ \t]+@?[A-Z!]/ {t++}
         /NOP[ \t]*;/ {n++}
         END {printf "%.2f%%\n", 100.0*n/t}'

The body-only count, run-length / predecessor / successor, and stall-
code distributions can each be produced from the same nvdisasm
output (or directly from raw cubin bytes):

• Body-only NOP fraction: walk each kernel's SASS, drop the
  trailing run of NOP-encoded instructions before counting. The
  encoding signature is (code & 0xFFF) == 0x918 on sm_120 (low-12
  bits of the 128-bit instruction).

• Stall-count histogram: each 16-byte SASS instruction carries a
  control word in its upper bits; the low 4 bits of that control word
  (bits 105..108 of the 128-bit code) are the stall count 0..15.
  Histogram those bits for instructions that decode to NOP.

• Run-length / predecessor / successor: scan each kernel; for
  each maximal run of NOP rows, record the preceding non-NOP opcode,
  the following non-NOP opcode, and the run length k. Histogram k
  and the opcode columns.

Per-cubin NOP fractions cross-checked between nvdisasm and
cuobjdump --dump-sass on a sample of 4 SHA-matched cubins (0 delta
on NOP counts).

Hardware / toolchain

• sm_80 target: standard A100/A40-class SM in the toolkit.
• sm_120 target: RTX 5080 (Blackwell consumer, GB203).
• CUDA toolkit: 13.1.
• Driver: 595.58.03 (build-time ptxas, not driver JIT).
• Disassembler: nvdisasm from CUDA 13.1.

What I'm proposing

Three options, your call:

1. Finding entry in FINDINGS.md under a new "Cross-arch
   observations" or "FP64 issue-rate scheduling" section: the tables,
   the controlled discriminator, and the OBS/INF/HYP block above.
2. Dedicated chapter (e.g. NN_blackwell_fp64_nop_density/)
   following the FINDINGS chapter convention: pick one cusolver or
   cublasLt kernel, dump sm_80 vs sm_120 SASS side-by-side, per-
   instruction-position NOP analysis with stall-count annotations,
   register which stall windows correlate with NOP placement against
   the GB203 FP64 dependent-latency budget.
3. No fit, close. If the cross-arch corpus angle isn't the kind
   of contribution sass-king is set up to receive, that's a fair
   answer.

I'd lean option 1 first; option 2 as a follow-up once a clean per-
kernel candidate is named.

Open gaps

• [GAP] --binary-kind=sass re-extraction not done. Would
  directly confirm NOPs are in the ptxas output rather than added
  downstream.
• [GAP] Producer-inclusive gap accounting is not yet run across
  the corpus. For each candidate dependency, include the predecessor and
  every intervening instruction, then join the producer's write-barrier
  to the consumer's wait mask. This is required before making a
  corpus-wide latency-reconstruction claim.
• [GAP] Per-instruction NCU profiling against a representative
  4-NOP gap is not yet measured; it would strengthen the issue-level
  interpretation independently of the paper comparison.
• [GAP] sm_89 (Ada) corpus not measured. Would isolate whether
  the effect began at Blackwell or earlier.
• [GAP] Other libraries (cudnn, cufft, NPP) not at corpus scale.
  cublasLt / cusolver / cusparse gives 3 points of the dense ↔ sparse
  axis; more would tighten the compute-density correlation.

Cross-reference

• FINDINGS.md ch.08 [OBS] "ptxas inserts 3 to 4 NOP between consecutive
  DADD instructions." This finding extends ch.08 from controlled
  instruction streams to corpus prevalence and adds an independent-chain
  mechanism discriminator.
• FINDINGS.md ch.11g [OBS] "NOP padding around DMUL confirms FP64
  throughput restriction." Same mechanism, observed across the
  cublasLt / cusolver / cusparse corpora.
• SASS King part 2 §3 (Huerta et al. 2025): the CGGTY control-code
  mechanism predicts exactly the bimodal stall distribution observed.
• arXiv:2507.10789v2 (Jarmusch, Graddon, and Chandrasekaran 2025):
  GB203 FP64 true/completion latency in Table III, with the separate
  two-instruction result discussed in the accompanying prose.

[florianmattana]

Thanks, this is a strong fit for SASS King. :)

This is a bit ahead of the current project phase: production-library audits are planned, but the public pattern/audit templates are not formalized yet. Since the project structure has just been reorganized around corpus/, knowledge/, and release-oriented onboarding, I want to integrate this carefully instead of dropping a large corpus-scale claim into the wrong place.

The finding extends existing ch.08 and ch.11g FP64/NOP observations cleanly, so I would rather accept it as the first corpus-scale external case and use it to shape the template.

I just think the reproducibility packet is the gate. Before promoting the corpus-scale claims to local SASS King [OBS], we would need:

A minimal reproducibility packet would be enough:

• one representative sm_120 cubin or SASS excerpt, plus the matching sm_80 artifact when available
• the scripts or commands used to compute body-only NOP rate, NOP-run predecessor/successor stats, and stall-code n histograms
• the exact cuobjdump / nvdisasm commands
• CUDA toolkit and driver versions.

Once attached, we will do the project-side bit-level validation: cross-check the NOP encoding signature (code & 0xFFF) == 0x918 against denvdis SM120 tables and the policy in knowledge/DENVDIS_INTEGRATION.md

Planned integration after the packet lands:

1. Add a short canonical entry in knowledge/FINDINGS.md, cross-referenced from ch.08 and ch.11g.
2. Keep the corpus-scale numbers as external evidence pending reproduction until at least one representative cubin is locally reproduced and bit-level checked.
3. Use this case to shape the upcoming pattern/audit layer, rather than forcing it into an existing controlled-kernel chapter.

So: yes, this is in scope. The next step is attaching the minimal reproducibility packet.

[sunnypatell]

Coming from the r/ReverseEngineering thread.

Your 14-cycle gap may just be the producer's own stall. The DFMA that starts the chain carries stall=15 itself, so it comes to 64 exactly with no pipeline tail:

DFMA R4, R4, R6, R4     stall=15  wr=1
NOP                     stall=15
NOP                     stall=15
NOP                     stall=15
NOP                     stall= 4
DFMA R4, R6, R4, R4     stall=15  wait=0x02

I measured DFMA/DADD at 63.99 cycles on a 5070 Ti, R² = 0.999999, consistent with the 63.57 GB203 figure you cite.

One caveat on reading latency out of NOP counts: the 64 is the chain cost, not the correctness requirement. The wr=1 / wait=0x02 scoreboard pair is what actually carries the dependency, and the NOP run is ptxas spending the latency inline rather than yielding. Detail in finding 8.

[delicious-pancakes]

Good catch. I omitted the producer DFMA's own stall=15 in the prose calculation. The direct chain is explicitly 15 + 15 + 15 + 15 + 4 = 64 cycles from producer issue to consumer issue, with no inferred pipeline tail. The wr=1 / wait=0x02 pair carries the long readiness dependency; the 64-cycle inline spacing is the chain cost and ptxas schedule, not the entire correctness requirement.

I-m correcting the write-up and will withdraw the 37.5-cycle histogram until I have a complete barrier-linked sequence for that case.