bench-profile

Reproducible CPU benchmarking environments for Linux, as boot profiles that survive kernel updates.

Three profiles are added to the boot menu, one per measurement question:

Profile	Answers	CPU state
single-core	"Is A faster than B?"	One isolated core, load balancing off, nothing else on the complex
multi-core	"How does it scale?"	A whole core complex as a real parallel pool, SMT off, boost off
unlimited	"How fast can this machine go?"	Every thread, SMT on, boost on, no isolation

This repository ships a configuration for an AMD Ryzen 9 9950X on CachyOS with the Limine bootloader. The topology values, unit names and bootloader integration are host-specific and live in one file, config/machine.conf. See Adapting to another CPU.

---

Motivation

Benchmark numbers are only comparable if the machine was in the same state each time. On a normal desktop boot it never is: the governor ramps, a browser wakes up, an IRQ lands on the core doing the work, the scheduler migrates your thread to a core with a cold L2. The differences you are trying to measure are often smaller than the noise these introduce.

The usual answer is to pin with taskset and hope. This project takes the stronger position that the machine should be in a known state before the benchmark starts, that the state should be asserted rather than assumed, and that the assertion should be cheap enough to run before every single run.

The failure that motivated the multi-core profile

An earlier version of this tooling had only one benchmark profile, built around isolcpus=domain,managed_irq,8-15. That is close to ideal for single-threaded A/B work. It is quietly catastrophic for parallel work.

isolcpus=domain removes those CPUs from every scheduler load-balancing domain. An affinity mask spanning 8-15 is then not enough to spread a thread pool: the kernel never migrates threads between isolated CPUs, so all eight workers pile onto whichever core the parent landed on and timeshare it.

Nothing errors. nproc still says 8. The thread pool still creates 8 threads. Throughput just looks like bad scaling. A full 29-cell benchmark matrix was collected over four hours against a machine that was silently running everything on one core:

f64, 2^24 points, 100k queries	16 cores, unpinned	isolated profile, "8 cores"
serial	1.61M q/s	1.83M q/s ✅
parallel	22.2M q/s (13.8× serial)	1.80M q/s (0.98×) ❌
a C++ competitor	13.1M q/s	0.86M q/s (1/15.2) ❌

Worse, the distortion was asymmetric. The Rust thread pool, which hands each worker a large contiguous chunk, lost only 2% to timesharing. The C++ competitor's finer-grained work-stealing scheduler spins while waiting, and spinning on a contended core is pure waste — it lost 52%. The resulting "speedup" numbers were inflated roughly 2× in one library's favour and would have been published.

Hence two things in this project:

• a separate multi-core boot profile whose isolcpus deliberately omits domain, keeping managed-IRQ isolation while leaving load balancing intact;
• a canary that proves work is actually spreading, rather than trusting the kernel command line to have said so.

---

The profiles

All three boot headless to multi-user.target with the desktop, networking, timers and other noisy units masked, and with swap disabled. They differ only in what they do to the CPU.

single-core

nosmt isolcpus=domain,managed_irq,8-15 nohz_full=8-15 rcu_nocbs=8-15
rcu_nocb_poll irqaffinity=0-7 systemd.cpu_affinity=0-7

The benchmark runs pinned to SINGLE_CORE_CPU (cpu8). Load balancing across 8-15 is off, which is exactly what you want here: the thread is bound to one CPU and must never migrate. The remaining benchmark CPUs are left idle and on the firmware governor — nothing runs there, so their frequency policy is irrelevant.

Boost stays on: one active core never exhausts the package power or thermal budget, so it contributes no drift, and it is part of the production- representative behaviour being measured.

Use this for A/B comparisons where you care about small differences.

multi-core

nosmt isolcpus=managed_irq,8-15 nohz_full=8-15 rcu_nocbs=8-15
rcu_nocb_poll irqaffinity=0-7 systemd.cpu_affinity=0-7

One token different from single-core — no domain — and that token is the entire point. managed_irq still keeps driver-steered interrupts off the benchmark CPUs; load balancing across them is preserved so thread pools spread.

bench-profile-multicore additionally:

• sets performance governor and EPP on every benchmark CPU (the single-core prep script only did cpu8, which is why seven of eight cores once sat on powersave during a parallel run);
• parks the idle complex (PARKED_CPUS, cpu1-7) so it drops into a deep C-state, handing its share of the package power and thermal budget to the benchmark cores and removing a source of fabric traffic. Housekeeping collapses onto cpu0. Use --keep-housekeeping-cores to A/B this;
• turns boost off by default. All-core boost clocks on a 16-core part track temperature and remaining power headroom, and that drift does not cancel between matrix cells measured minutes apart — a later cell can read slower purely because the part is hotter. Losing some absolute throughput to make cells comparable is the right trade here. Override with --boost=on;
• runs the canary, and refuses to report success if it fails.

The benchmark CPUs must share one L3 domain; the status tool checks this, because straddling two complexes silently mixes cross-complex traffic into results.

unlimited

(no CPU restrictions at all)

Every thread, SMT on, boost on, no isolation, no pinning, no IRQ steering. This measures the ceiling: the most throughput the part can actually deliver, at the cost of contending with housekeeping.

SMT defaults on, and that is a deliberate choice rather than merely "no restrictions". Workloads that stall on dependent, hard-to-prefetch loads — tree traversals, pointer chasing, most search structures — are exactly where a second thread per core fills otherwise-idle issue slots. Use --smt=off to measure the contribution rather than assume it.

There is no IRQ guard in this profile. Housekeeping shares these cores by design; that is the trade being made, not a defect.

---

Usage

Install

just install          # or: sudo ./install

This installs the tools to /usr/local/bin, the library and machine config to /usr/local/lib/bench-profile/, the boot hook to /etc/boot/hooks/post.d/95-add-benchmark-boot-entries, and bench-prep.service to /etc/systemd/system/. It then regenerates the boot entries.

Everything replaced is copied to /var/backups/bench-profile/<timestamp>/ first, including the bootloader config. Files from an older single-profile install are renamed .superseded rather than deleted.

Use just install-keep-config on subsequent installs to preserve a machine.conf you have customised for the host.

Boot and verify

Reboot and pick an entry ending in benchmark single-core, benchmark multi-core or benchmark unlimited. bench-prep.service applies the matching policy automatically before multi-user.target.

Then assert the state:

bench-profile-status expect-multi-core

Exits 1 with an explanation if anything is off, including being booted into the wrong profile. Wire this into your benchmark harness as a gate.

Report without failing

bench-profile-status          # ALWAYS exits 0

The bare form is a report, not an assertion. It prints an observed- environment block (host, kernel, CPU model, online/isolated CPUs, SMT, boost, governors, masks) followed by every check, with shortfalls as WARN rather than FAIL. It never exits non-zero.

That contract exists so it can be embedded in unattended runs on any boot — including a normal desktop boot — and its output logged next to the results, without ever failing the job it is only documenting:

bench-profile-status > results/environment.txt
cargo bench >> results/bench.log

Run with an IRQ audit

bench-profile-run cargo bench

Validates the environment, pins the command tree to the profile's benchmark CPUs, and samples their numbered hardware-IRQ counters around the whole run. If any device IRQ arrives, the measurement is invalid and the wrapper exits 125 — deliberately distinct from the command's own status, so a harness can detect and retry it:

until bench-profile-run cargo bench; do
    (( $? == 125 )) || break     # 125 means "interference, retry"
done

Inter-processor interrupts are excluded from the guard: ordinary process wakeups legitimately produce them, so counting them would reject every run.

Everyday tasks

just                          # list all tasks
just test                     # full suite; needs no root, safe on any boot
just show-config              # effective config, including derived cpumasks
just canary                   # prove work spreads across the benchmark CPUs
just which                    # which profile is booted
just preview-boot-entries     # see what would be generated, without writing
just regenerate-boot-entries  # re-run the hook by hand

---

Effects

What each tool changes

bench-prep dispatches on the profile marker to one of three prep scripts. Between them they set:

Setting	single-core	multi-core	unlimited
scaling_governor / EPP performance	cpu8	all benchmark CPUs	every online CPU
cpufreq/boost	on	off	on
SMT	off (boot)	off (boot)	on
Unbound workqueue cpumask	housekeeping	housekeeping	untouched
default_smp_affinity + movable IRQs	housekeeping	housekeeping	untouched
watchdog_cpumask	housekeeping	housekeeping	untouched
Idle complex parked	no	yes	no
numa_balancing, sched_autogroup_enabled	0	0	0
vm/stat_interval	120	120	120
perf_event_paranoid, kptr_restrict	relaxed	relaxed	relaxed
swap	off	off	off

The workqueue and IRQ masks are not narrowed in the unlimited profile: constraining them would starve exactly the cores that profile is trying to saturate.

perf_event_paranoid and kptr_restrict are relaxed only because these boots are headless, single-user and offline. They make perf annotate usable without weakening the normal boot profile.

What is deliberately left alone

Turbo (except in multi-core, and there only for cross-cell stability), transparent huge pages, ASLR, CPU mitigations and normal idle states all remain enabled. They are part of the production-representative CPU behaviour being measured; disabling them would change the comparison rather than merely reduce noise.

Immutable IRQ affinities

NVMe and similar drivers use affinity-managed per-CPU queues whose masks cannot be rewritten. Their effective affinity can therefore still name a benchmark CPU even though isolcpus=managed_irq stops housekeeping I/O from selecting the isolated queue. The tools report these as notes, never failures — the authority is the measured hardware-IRQ delta, checked both when validating the profile and around each audited run.

Boot-entry regeneration

A kernel update rewrites limine.conf from scratch, discarding anything added by hand. limine-entry-tool runs the hooks in /etc/boot/hooks/post.d afterwards, so 95-add-benchmark-boot-entries re-adds the entries every time, unattended.

It finds the OS entry for this machine (by machine-id comment, falling back to matching kernel paths), clones every non-fallback kernel entry matching KERNEL_ENTRY_MATCH once per profile, merges the profile cmdline into each, and wraps the result in # BEGIN/END AUTO BENCHMARK ENTRIES markers. It strips its own previous output first, so it is idempotent, and it also removes entries left by the older single-profile hook. The previous limine.conf is saved to limine.conf.bench.bak.

The entries are inserted before any Snapshots block so they stay near the top of the menu.

---

The canary

The check that would have caught the four-hour mistake.

Parallel-dispatch canary on cpu8-15 (8 workers, need >=4x)...
  one worker:        297 ms (8000000 iterations)
  8 workers at once: 304 ms
  workers seen on 8 distinct CPUs: 8 9 10 11 12 13 14 15
  effective parallelism: 7.81x of 8x
  canary passed: work spreads across the benchmark CPUs.

It runs one CPU-bound spinner to calibrate, then N at once, and compares wall clock. Real parallelism means N spinners finish in about the time of one; a collapsed pool takes N times as long. It also samples /proc/PID/stat to report how many distinct CPUs the workers actually landed on.

This tests the property that actually matters and is indifferent to which threading library the benchmark uses — it catches the problem for rayon, ParlayLib, OpenMP or anything else.

Failure names the usual cause:

CANARY FAILED: the benchmark CPUs are not dispatching work in parallel.
  8 concurrent workers took 2293 ms; one alone took 292 ms.
  That is below the 4x floor, so threads are piling onto too few cores.
Almost always this means the kernel cmdline carries isolcpus=domain over 8-15...

The threshold is half the available parallelism (CANARY_SPEEDUP_DIVISOR), which is deliberately loose: the canary exists to catch a pool collapsing onto one core, not to grade scaling efficiency. In the unlimited profile the floor is computed from physical cores, not logical ones — an ALU-bound probe gains almost nothing from SMT (measured: 15.0× from 32 threads on 16 cores), so gating on logical CPUs would false-fail.

---

Testing

just test

Needs no root, no reboot and no benchmark profile; it touches no system state. Five suites:

Suite	Covers
test-lib	expand_cpu_list, count_cpus, cpulist_to_mask (including multi-word masks), mask normalisation, plus internal consistency of the shipped machine.conf — 28 assertions
test-boot-hook	Generation against synthetic limine.conf files: entry counts, per-profile cmdline correctness (asserting multi-core has no domain), idempotency, ordering before Snapshots, kernel_path fallback matching, migration from the legacy hook, and that an unmatched KERNEL_ENTRY_MATCH fails loudly while leaving the config untouched
test-status	The exit-code contract: the bare form always exits 0 and emits no FAIL lines, expect-* forms exit 1 off-profile, unknown arguments exit 2, the environment report contains the fields worth archiving
test-canary	Both directions — passes when work spreads across four CPUs, fails when four workers are confined to one
test-syntax	bash -n on every script, config sourceability, executable bits, and shellcheck -S warning if installed

The boot-hook tests work on synthetic copies in a temp directory and never touch /boot.

What the suite cannot cover: anything requiring an actual reboot into a profile. The prep scripts' effects, bench-prep.service firing, and IRQ isolation are only verifiable by booting. The status tool is the check for those, which is why it exists in assertion form.

---

Adapting to another CPU

Everything host-specific is in config/machine.conf. Nothing is hardcoded in the tools; cpumasks are derived from CPU lists rather than maintained as hex constants, so the two cannot disagree.

The shipped values describe an AMD Ryzen 9 9950X: Zen 5, 16 cores / 32 threads, two CCDs of 8 cores, each with its own 32MB L3. Linux enumerates the primary threads as 0-15 and the SMT siblings as 16-31, so CCD1's primary threads are 8-15.

Choosing BENCH_CPUS

The benchmark set should be one cache complex, so results are not polluted by cross-complex traffic. Find the complexes:

cat /sys/devices/system/cpu/cpu*/cache/index3/shared_cpu_list | sort -u
lscpu --extended=CPU,CORE,SOCKET,CACHE

List primary threads only — SMT siblings are taken offline by nosmt. Check which half of a sibling pair is primary:

cat /sys/devices/system/cpu/cpu8/topology/thread_siblings_list   # e.g. "8,24"

Then set:

BENCH_CPUS="8-15"          # one complex, primary threads
HOUSEKEEPING_CPUS="0-7"    # everything else
SINGLE_CORE_CPU=8          # must be inside BENCH_CPUS
PARKED_CPUS="1-7"          # housekeeping CPUs to offline in multi-core; never cpu0
EXPECTED_ONLINE_CPUS="0-15" # what `nosmt` should leave online

just test validates these are self-consistent (no overlap, SINGLE_CORE_CPU inside BENCH_CPUS, PARKED_CPUS a subset of housekeeping and excluding cpu0), and just show-config prints the derived masks.

Single-CCD, Intel, and other topologies

• One cache complex only (most Intel desktop parts, Zen with a single CCD): split the cores rather than the complexes — e.g. BENCH_CPUS="4-7", HOUSEKEEPING_CPUS="0-3". You lose the "private L3" property, so expect the multi-core profile to be noisier; the L3-domain check in bench-profile-status will still pass because the whole set shares one L3.
• Intel P/E-core hybrids: put BENCH_CPUS entirely on P-cores or entirely on E-cores, never a mix. lscpu --extended shows the core types.
• Multi-socket / NUMA: keep the set within one NUMA node, and consider pinning memory with numactl --membind.
• No SMT: drop nosmt from the profile cmdlines in boot/95-add-benchmark-boot-entries and set EXPECTED_ONLINE_CPUS to all CPUs.

Distribution and desktop

MASKED_UNITS and NOISY_UNITS are CachyOS + KDE. Masking a unit that does not exist is harmless, so pruning is optional, but replace the desktop entries (plasmalogin.service, ly@tty2.service) with your display manager, and drop CachyOS-specific ones (cachyos-iw-set-regdomain.path, scx_loader.service, ananicy-cpp.service, bpftune.service) if absent. Find candidates with:

systemd-analyze blame | head -30
systemctl list-units --state=running

Bootloader

The boot hook targets Limine, driven by limine-entry-tool's /etc/boot/hooks/post.d mechanism, and understands Limine's nested /OS → //Kernel config syntax.

For systemd-boot or GRUB the cloning logic in boot/95-add-benchmark-boot-entries must be rewritten — the profile cmdlines and the marker convention carry over unchanged, but the parsing does not. Whatever replaces it needs to run after kernel updates: a systemd-boot version would drop entries into /boot/loader/entries/, and a GRUB version would write a /etc/grub.d/ snippet. KERNEL_ENTRY_MATCH selects which kernels get cloned and is the only bootloader-adjacent value in machine.conf.

After changing the config

just test                     # validates consistency
just preview-boot-entries     # see the generated cmdlines without writing
just install                  # apply, and regenerate the entries

Then reboot and bench-profile-status expect-<profile>.

---

Layout

bin/     bench-prep (dispatcher), the three prep scripts,
         bench-profile-status, bench-profile-run
lib/     bench-profile-common.sh — shared helpers and the canary
boot/    95-add-benchmark-boot-entries — the limine-entry-tool post-hook
systemd/ bench-prep.service
config/  machine.conf — everything host-specific
tests/   run-all plus five suites
install  installer, with backups
justfile task runner

Requirements

Bash 4.4+, python3, taskset and awk (util-linux and any awk), systemd. just for the task runner and shellcheck for linting are both optional.