Skip to content

feat(core): add experimental performance report - #6136

Open
lo1tuma wants to merge 1 commit into
stryker-mutator:masterfrom
lo1tuma:experimental-performance-report
Open

feat(core): add experimental performance report#6136
lo1tuma wants to merge 1 commit into
stryker-mutator:masterfrom
lo1tuma:experimental-performance-report

Conversation

@lo1tuma

@lo1tuma lo1tuma commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Adds an opt-in, experimental performance report to help diagnose and compare mutation-testing performance. This is the groundwork for measuring the static-mutant performance work discussed in #6135.

Enable it with experimentalPerformanceReport: true, and Stryker writes reports/mutation/performance.json alongside the normal reports. It captures per-phase timing (setup, initial run, mutation, reporting), the static-vs-runtime cost split, environment reloads and reload time, per-worker utilization, retry/out-of-memory counts, and machine and configuration info. The standard mutation.json and HTML report are untouched, and there is no behavior change unless the option is set.

The format is intentionally experimental and unstable: the idea is to try it as a dedicated, opt-in channel first and, if it proves useful, consider promoting parts of it into the standard report later. docs/performance-report.md documents every field and, importantly, how to read the numbers: wall-clock phases (which sum to the total) versus work summed across parallel workers.

I ran this on a large real-world mocha + TypeScript project (~5500 mutants). Beyond the well-known #3282 pattern (static mutants are a small fraction of the count but a large fraction of the cost), the metrics surfaced things that weren't visible before: reloadEnvironmentLast doesn't order static plans globally, so static and runtime work interleaves and each interleaving triggers an extra full-restart reload on the following runtime mutant. That adds up to hundreds of avoidable reloads, which is why totals.reload reports more reloads than there are static mutants. Pure environment reload accounted for ~23% of static-mutant time, and ~45% of mutants were killed as CompileError by the type checker before ever reaching the test runner.

Closes #6135

Add an opt-in performance report, enabled with the
experimentalPerformanceReport option, that writes
reports/mutation/performance.json with per-phase timing, the
static-vs-runtime cost split, environment reloads and reload time,
per-worker utilization, and retry and out-of-memory counts. The standard
reports are untouched and there is no behavior change unless the option is
set.

The format is experimental and unstable, documented in
docs/performance-report.md.

Closes stryker-mutator#6135
lo1tuma added a commit to lo1tuma/stryker-js that referenced this pull request Jul 17, 2026
Static mutants force a test-runner environment reload. Plans were sorted only
within each buffered batch, so static and non-static plans interleaved and every
interleaving forced an extra reload on the non-static mutant that followed a
static one. Ordering the whole run non-reload-first and reload (static) last
removes those transition reloads, while non-static mutants still stream and
overlap with checking as before.

The effect is measurable with the experimental performance report (stryker-mutator#6136):
totals.reload drops toward the static-mutant count, which on static-heavy suites
using full-restart runners (mocha, jasmine, karma, tap) cuts wall time.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose a timing breakdown (per phase + static vs runtime) so performance work is measurable

1 participant