feat(core): add experimental performance report - #6136
Open
lo1tuma wants to merge 1 commit into
Open
Conversation
Add an opt-in performance report, enabled with the experimentalPerformanceReport option, that writes reports/mutation/performance.json with per-phase timing, the static-vs-runtime cost split, environment reloads and reload time, per-worker utilization, and retry and out-of-memory counts. The standard reports are untouched and there is no behavior change unless the option is set. The format is experimental and unstable, documented in docs/performance-report.md. Closes stryker-mutator#6135
lo1tuma
added a commit
to lo1tuma/stryker-js
that referenced
this pull request
Jul 17, 2026
Static mutants force a test-runner environment reload. Plans were sorted only within each buffered batch, so static and non-static plans interleaved and every interleaving forced an extra reload on the non-static mutant that followed a static one. Ordering the whole run non-reload-first and reload (static) last removes those transition reloads, while non-static mutants still stream and overlap with checking as before. The effect is measurable with the experimental performance report (stryker-mutator#6136): totals.reload drops toward the static-mutant count, which on static-heavy suites using full-restart runners (mocha, jasmine, karma, tap) cuts wall time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in, experimental performance report to help diagnose and compare mutation-testing performance. This is the groundwork for measuring the static-mutant performance work discussed in #6135.
Enable it with
experimentalPerformanceReport: true, and Stryker writesreports/mutation/performance.jsonalongside the normal reports. It captures per-phase timing (setup, initial run, mutation, reporting), the static-vs-runtime cost split, environment reloads and reload time, per-worker utilization, retry/out-of-memory counts, and machine and configuration info. The standardmutation.jsonand HTML report are untouched, and there is no behavior change unless the option is set.The format is intentionally experimental and unstable: the idea is to try it as a dedicated, opt-in channel first and, if it proves useful, consider promoting parts of it into the standard report later.
docs/performance-report.mddocuments every field and, importantly, how to read the numbers: wall-clock phases (which sum to the total) versus work summed across parallel workers.I ran this on a large real-world mocha + TypeScript project (~5500 mutants). Beyond the well-known #3282 pattern (static mutants are a small fraction of the count but a large fraction of the cost), the metrics surfaced things that weren't visible before:
reloadEnvironmentLastdoesn't order static plans globally, so static and runtime work interleaves and each interleaving triggers an extra full-restart reload on the following runtime mutant. That adds up to hundreds of avoidable reloads, which is whytotals.reloadreports more reloads than there are static mutants. Pure environment reload accounted for ~23% of static-mutant time, and ~45% of mutants were killed asCompileErrorby the type checker before ever reaching the test runner.Closes #6135