perf: measure the samples in parallel jobs (#60)

* perf: measure the samples in parallel jobs and publish from one that measures nothing

The workflow ran `pnpm run benchmark` three times in a row in a single job:
8m05s, 7m57s, 8m05s, for 24m20s of a run whose measurements take eight
minutes. The three are independent — each makes its own temp directory,
provisions its own package managers, and starts its own registry on a port
the system says is free, and the only thing a later one reads from an
earlier is the sample count that `LIMIT_RUNS` caps, which three samples
never reach. So they now run as a matrix.

Three samples rather than one because the page reports the minimum per
package manager *per version*. Cutting to a single sample wouldn't only add
noise, it would bias: a version released this week would be a min-of-one
while a long-lived one keeps its min-of-many, which reads as a regression in
whatever was released most recently.

Splitting the measuring means something has to join it back together, so
`index.js` grows a `--report-only` mode. `benchmarkFixtures` and
`nodeVersionsSection` now take a callback for one manager's results —
measuring measures, reporting reads back what a measuring run recorded — and
everything downstream of that only draws, so there is one copy of it. A
reporting run installs no package manager and starts no registry, which is
what keeps it at seconds rather than minutes, and it means the job that
commits results can't fail for a benchmarking reason. It also can't invent a
number a measuring run failed to record: it fails instead.

What it can't work out for itself is which version of each tool the results
were measured with, since nothing is installed to ask — hence the
`versions.json` a measuring run writes and its job uploads. The Node.js
version goes in it too: the charts say "Tests were run using Node.js X", and
that is the run that measured them, not the one drawing the page.

`mergeResults.js` folds the runs together. Each starts from the same commit
and appends to the results it checked out, so what a run contributes is
whatever it added past the length of the baseline — its whole file still
carries every sample recorded before this week and would otherwise be
counted once per job.

Verified by rebuilding the published page from the committed results alone:
identical tables and numbers, differing only in the timestamp. And by
merging three simulated runs against the real results tree: three samples
added, one per run, a version no baseline had getting exactly three, and
only the file that actually changed showing up in `git status`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: fail a reporting run on a manifest it can't trust, and keep the measuring jobs read-only

Two review findings on the parallel-jobs change.

The workflow used to be one job, and that job declared its own permissions.
Splitting it left the measuring jobs inheriting whatever the repository
defaults to, so the default is now read-only and the one job that publishes
raises it for itself.

The reporting run caught a manifest it couldn't read or parse, but not one
that parsed into the wrong shape. A missing package manager would surface
downstream — the results aren't where its version says they are — but a
missing pnpr version surfaced nowhere: the page went out saying the registry
everything was measured against was `vundefined`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: demand a version for the Node.js section's tools too

The manifest guard checked every package manager column but only that the
`nodeManagers` object existed, so an empty or partial one got as far as
`readRecordedResults`. That did name the tool it couldn't place, but by then
the fixture section had already been built, and nothing said the manifest
was what was wrong.

Checked over the keys of `nodeManagersMap`, which is what
`writeVersionsManifest` writes the section from, so what a reporting run
demands is exactly what a measuring run records.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: zkochan

.github/workflows/benchmark.yml

@@ -5,13 +5,33 @@ on:
     - cron: '30 2 * * 0'
   workflow_dispatch:
 
+# Read-only by default, so the jobs that only measure cannot write to the
+# repository whatever the repository's own default happens to be. The one job
+# that publishes raises this for itself.
+permissions:
+  contents: read
+
 jobs:
-  build:
+  # The samples this run contributes, measured in parallel rather than one
+  # after another. Three measurements of the same thing are what the published
+  # number is a `min()` of, and they are independent of each other: each makes
+  # its own temp directory, provisions its own package managers, and starts its
+  # own registry on a port the system says is free. Run sequentially in one job
+  # they took three times as long as a single measurement for no benefit.
+  #
+  # Three samples rather than one because the page reports the minimum per
+  # package manager *per version*. Cutting to one sample wouldn't only add
+  # noise, it would bias: a version released this week would be a min-of-one
+  # while a long-lived one keeps its min-of-many, which reads as a regression
+  # in whatever was released most recently.
+  benchmark:
+    strategy:
+      # One machine having a bad day shouldn't throw away the samples the
+      # others already measured. The reporting job publishes whatever arrived.
+      fail-fast: false
+      matrix:
+        sample: [1, 2, 3]
     runs-on: ubuntu-latest
-    # The job commits the results it measures back to the branch, which the
-    # default read-only token can't do.
-    permissions:
-      contents: write
     steps:
       - uses: actions/checkout@v4
         with:
@@ -37,12 +57,46 @@ jobs:
       - run: pnpm run benchmark
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      - run: pnpm run benchmark
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-      - run: pnpm run benchmark
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      # Only uploaded when the measurement finished. A run that died partway
+      # recorded some scenarios and not others, and those are exactly the
+      # numbers not to publish.
+      - uses: actions/upload-artifact@v4
+        with:
+          name: samples-${{ matrix.sample }}
+          path: |
+            results
+            versions.json
+          retention-days: 3
+
+  # Draws the page from the samples the jobs above recorded. It measures
+  # nothing itself — no package manager is installed here and no registry is
+  # started — so it can't quietly substitute a number for one a measuring run
+  # failed to record, and it costs seconds rather than minutes.
+  report:
+    needs: benchmark
+    # `needs` alone would skip this job if any one sample failed, throwing away
+    # the two that succeeded.
+    if: ${{ !cancelled() }}
+    runs-on: ubuntu-latest
+    # The job commits the results it publishes back to the branch, which the
+    # default read-only token can't do.
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+          fetch-depth: 0
+      - uses: pnpm/setup@v2
+      # Fails when no sample arrived at all, which is the right outcome: with
+      # nothing measured this week, the alternative is republishing last week's
+      # numbers under today's date.
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: samples-*
+          path: samples
+      - run: pnpm run merge-results samples
+      - run: pnpm run report
       - name: Commit & Push changes
         # Pinned: this step is handed a token that can write to the repo, so
         # what runs in it shouldn't be able to change without a commit here.

.gitignore

@@ -5,3 +5,5 @@ node_modules
 .tmp
 
 managers
+
+versions.json

generateStackedSvg.js

@@ -4,7 +4,8 @@
 
 const getMax = (results) => Math.max(...results.map((r) => r.v11))
 
-export default (results, formattedNow) => {
+// `nodeVersion`: see the note in `generateSvg.js`.
+export default (results, formattedNow, nodeVersion = process.version) => {
   const v12Color = '#fbae00'
   const extraColor = '#cccccc'
 
@@ -102,7 +103,7 @@ export default (results, formattedNow) => {
     })
   })
 
-  svgStr += `  <text x="${graph.x + graph.w}" y="${vb.h - 2}" class="font s4 text" text-anchor="end">Tests were run using Node.js ${process.version} at: ${formattedNow}</text>\n`
+  svgStr += `  <text x="${graph.x + graph.w}" y="${vb.h - 2}" class="font s4 text" text-anchor="end">Tests were run using Node.js ${nodeVersion} at: ${formattedNow}</text>\n`
 
   svgStr += '</svg>\n'
   return svgStr

generateSvg.js

@@ -17,7 +17,9 @@ const getHighestNumber = (resultArrays) => {
   return max
 }
 
-export default (resultArrays, pms, tests, formattedNow) => {
+// `nodeVersion` is the Node.js the results were measured on. A reporting run
+// draws a page it did not measure, so it can't use its own `process.version`.
+export default (resultArrays, pms, tests, formattedNow, nodeVersion = process.version) => {
   let svgStr = ''
   // Expand stacked PM slots into two legend entries (primary + extra).
   const legendEntries = pms.flatMap((pm) => pm.stacked
@@ -217,7 +219,7 @@ export default (resultArrays, pms, tests, formattedNow) => {
 
   // add node version
   ;(() => {
-    const text = `Tests were run using Node.js ${process.version} at: ${formattedNow}`
+    const text = `Tests were run using Node.js ${nodeVersion} at: ${formattedNow}`
     const anchor = 'end'
     const x = graph.x + graph.w
     const y = vb.h - 2