Test Impact Analysis

[sebastianbergmann]

The changes proposed here implement test impact analysis: PHPUnit records which source files each test depends on, and can then run only the tests a change can affect.

Closes #6897.

What it does

Recording

--record-test-impact-data, or recordTestImpactData="true", records while the tests run which source files each test depends on. The data goes into its own file in the cache directory, separate from the test run history.

Querying

--list-tests-that-depend-on src/Money.php queries what was recorded, without running a single test:

Recorded from what the tests executed.

Tests that depend on /…/src/Money.php as it is now:
 - Example\InvoiceTest::testTotalIsTheSumOfItsItems
 - Example\SumTest::testAdds#0

Tests that depend on an earlier version of /…/src/Money.php:
 - Example\FormatterTest::testFormatsAmount

The two lists are kept apart on purpose: the first describes the code as it is now, the second describes tests recorded against a version of that file which no longer exists.

Selecting

--only-impacted runs only the tests a change can affect:

Impact:        26 of 177 tests can be affected by what changed; 151 tests are not run

OK (26 tests, 80 assertions)

The line says how many were left out, and the summary reports the tests that actually ran. Nothing pretends the other 151 passed.

PHPUnit works out what changed by comparing what is there now against what it recorded. --impacted-by src/Money.php --impacted-by src/Service tells it instead and it implies --only-impacted. What you name is the change set: the recorded hashes are not consulted at all. That matters on a fresh checkout, where comparing against the recording reports everything that changed since, while git diff --name-only reports what you actually did.

Because naming paths one at a time does not compose with the tool that knows what changed, --impacted-by-file reads a list, one path per line, - being standard input:

$ git diff --name-only | phpunit --impacted-by-file -

An empty list is an answer rather than a missing one: nothing changed, so nothing that depends on code runs. A list that cannot be read is not an empty list, and PHPUnit stops rather than guessing.

Two ways of maintaining the data

Observation needs code coverage data collection and works no matter what a project declares. --derive-test-impact-data-from-coverage-targets is the alternative for projects that already take coverage targets seriously: it works out what each test depends on from the #[CoversClass] and #[UsesClass] attributes it declares, without code coverage data collection, and without the tests having been run even once. beStrictAboutCoverageMetadata is what makes it sound: under it, a test that executes code it does not declare is already marked risky, so declarations are a superset of execution. PHPUnit warns when the mode is used without the strict check, because then nothing has ever verified the declarations.

Measurements

All from one real project of 177 tests that enables both requireCoverageMetadata and beStrictAboutCoverageMetadata.

	wall clock
the suite, no coverage driver	5.44s
recording derived from declarations	5.47s
a coverage run, no recording	33.5s
the same coverage run, recording	87.8s

Recording by observation costs 2.6× on a run that was already collecting coverage, and all of it lands in the tests declaring #[CoversNothing], which have to be collected for and are usually the slow ones. Deriving from declarations is free.

For 165 of 165 tests that declare targets, the declarations covered everything the test was observed to execute — no exceptions beyond #[CoversNothing]. The price is a coarser selection:

changed file	from what tests executed	from what tests declare
the matrix class	129 of 177	138
the tuple class	123	164
the world class	16	25
an output mapper	6	6

Declaration is never below observation on any file, and costs between nothing and half as many tests again. Five of the 177 tests declare no targets, are never recorded in that mode, and therefore always run: that is the floor.

Selecting after editing two source files gave 26 of 177 whether the change set was worked out from the hashes or piped in from git diff --name-only. With nothing edited, --only-impacted selected 0 of 177 in 0.14 seconds. The data is about 150 to 350 bytes per test — 27 KiB for this project.

What you can rely on

Everything the analysis has no reliable information about causes the test to run:

• a test that was never recorded, including a test declaring #[CoversNothing] and a PHPT test
• a test whose own code changed, or whose data provider changed
• a test that did not pass when it was last run
• a test another selected test depends on through #[Depends]

And these cause every test to run:

• nothing has been recorded yet, or what was recorded is no longer usable
• a source file changed that no test is recorded as depending on
• a source file is there that was not there, or was not first-party code, when the recording was made
• a path that was named, with --impacted-by or in a list, is not among the files that were recorded
• the configuration file changed, the set of first-party code changed, or composer.lock changed

The last one matters more than it sounds: what was recorded describes one state of the world, and when that state is not what it was, the honest answer is not that some entries are stale but that none of them can be trusted.

--only-impacted refuses to run at all without a cache directory, or when the test run history is turned off, rather than quietly being less careful.

Nothing is kept forever, either. A test run forgets every entry it did not record itself, but only when that run ran every test there is. A run that filtered, that selected by impact, or that stopped at the first failure recorded nothing for the tests it did not run, and nothing tells those apart from tests that are gone, so such a run adds to what is known and takes nothing away.

#[UsesFixture]

Neither observation nor declaration can see a data file: reading invoices.csv is not executing code, and a coverage target names code units. A new attribute closes that:

#[UsesFixture('../fixtures/invoices.csv')]
public static function provideInvoices(): array

It goes on a test class, a test method, or a data provider method: the last is the one that earns its keep, because a provider shared by many tests declares the file once and every test using it inherits the dependency. Directories work too, watched as a whole, so adding a file to one counts as a change. A path that does not exist produces a warning and is ignored, the same treatment a coverage target that cannot be used gets. Unlike coverage targets, nothing can ever verify this attribute; it is a promise the project makes to itself.

Decisions

• Recording is never automatic, not even when coverage is already being collected. A suite whose slow tests declare #[CoversNothing] would otherwise get 2.6× slower unasked.
• Tests declaring #[CoversNothing] are collected for in observation mode, with the targets bypassed so nothing reaches the coverage report. The alternative — treating them as affected by everything — would put a floor under every selected run.
• Impact data is never filtered using coverage targets; the report always is. The two disagree by design. Before this, what the coverage report kept was missing 75% and 94% of the dependencies tests really have on the two suites I measured, because the report is narrowed to what a test declares. That is why the data is collected before the narrowing, which needed php-code-coverage 14.4.0.
• The data is per-machine and must not be committed. It holds absolute paths and is discarded on a PHPUnit or PHP version change.
• A test's own files are recorded in the impact data rather than checked against the test index at selection time. The index is written at a different moment, so a run that refreshed the index without recording impact data would report a changed test as unchanged.
• The assumption hash covers the <source> declarations, not the expanded file list. Hashing the list would throw everything away each time a class is added to src/, and buys nothing: a new file can only be reached if an existing one changed.
• --only-impacted, --impacted-by and --impacted-by-file are command line options only. A switch that makes every run partial does not belong in a file a project commits.

[github-actions]

API Surface Changes

If any of the additions below are not intended as public API, mark them with @internal in the docblock.

New API Surface

Classes

• PHPUnit\Framework\Attributes\UsesFixture — class PHPUnit\Framework\Attributes\UsesFixture
• PHPUnit\Metadata\UsesFixture — class PHPUnit\Metadata\UsesFixture extends PHPUnit\Metadata\Metadata

Methods

• PHPUnit\Framework\Attributes\UsesFixture::__construct — public function __construct(string $path)
• PHPUnit\Framework\Attributes\UsesFixture::path — public function path(): string
• PHPUnit\Metadata\Metadata::usesFixtureOnClass — static public function usesFixtureOnClass(string $path): PHPUnit\Metadata\UsesFixture
• PHPUnit\Metadata\Metadata::usesFixtureOnMethod — static public function usesFixtureOnMethod(string $path): PHPUnit\Metadata\UsesFixture
• PHPUnit\Metadata\Metadata::isUsesFixture — public function isUsesFixture(): bool
• PHPUnit\Metadata\MetadataCollection::isUsesFixture — public function isUsesFixture(): self
• PHPUnit\Metadata\UsesFixture::path — public function path(): string
• PHPUnit\TextUI\Configuration\Configuration::recordTestImpactData — public function recordTestImpactData(): bool
• PHPUnit\TextUI\Configuration\Configuration::deriveTestImpactDataFromCoverageTargets — public function deriveTestImpactDataFromCoverageTargets(): bool

Modified API Surface

Methods

• PHPUnit\TextUI\Configuration\Configuration::__construct

  - public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $recordTestRunHistory, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testRunHistoryFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, bool $coverageHtmlClassView, bool $coverageHtmlFileView, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageJsonl, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnSelfDeprecation, bool $failOnDirectDeprecation, bool $failOnIndirectDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnSelfDeprecation, bool $doNotFailOnDirectDeprecation, bool $doNotFailOnIndirectDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomOrderSeed, int $repeat, int $retry, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold, bool $warnWhenPhpIsNotConfiguredForDevelopment, bool $cacheTestIndex)
  + public function __construct(array $cliArguments, ?string $testFilesFile, ?string $configurationFile, ?string $bootstrap, array $bootstrapForTestSuite, bool $recordTestRunHistory, ?string $cacheDirectory, ?string $coverageCacheDirectory, Source $source, string $testRunHistoryFile, ?string $coverageClover, ?string $coverageCobertura, ?string $coverageCrap4j, int $coverageCrap4jThreshold, ?string $coverageHtml, bool $coverageHtmlClassView, bool $coverageHtmlFileView, int $coverageHtmlLowUpperBound, int $coverageHtmlHighLowerBound, string $coverageHtmlColorSuccessLow, string $coverageHtmlColorSuccessLowDark, string $coverageHtmlColorSuccessMedium, string $coverageHtmlColorSuccessMediumDark, string $coverageHtmlColorSuccessHigh, string $coverageHtmlColorSuccessHighDark, string $coverageHtmlColorSuccessBar, string $coverageHtmlColorSuccessBarDark, string $coverageHtmlColorWarning, string $coverageHtmlColorWarningDark, string $coverageHtmlColorWarningBar, string $coverageHtmlColorWarningBarDark, string $coverageHtmlColorDanger, string $coverageHtmlColorDangerDark, string $coverageHtmlColorDangerBar, string $coverageHtmlColorDangerBarDark, string $coverageHtmlColorBreadcrumbs, string $coverageHtmlColorBreadcrumbsDark, ?string $coverageHtmlCustomCssFile, ?string $coverageJsonl, ?string $coverageOpenClover, ?string $coveragePhp, ?string $coverageText, bool $coverageTextShowUncoveredFiles, bool $coverageTextShowOnlySummary, ?string $coverageXml, bool $coverageXmlIncludeSource, bool $pathCoverage, bool $branchCoverage, ?string $coverageDriver, bool $ignoreDeprecatedCodeUnitsFromCodeCoverage, bool $disableCodeCoverageIgnore, bool $disableCoverageTargeting, bool $failOnAllIssues, bool $failOnDeprecation, bool $failOnSelfDeprecation, bool $failOnDirectDeprecation, bool $failOnIndirectDeprecation, bool $failOnPhpunitDeprecation, bool $failOnPhpunitNotice, bool $failOnPhpunitWarning, bool $failOnEmptyTestSuite, bool $failOnIncomplete, bool $failOnNotice, bool $failOnRisky, bool $failOnSkipped, bool $failOnWarning, bool $doNotFailOnDeprecation, bool $doNotFailOnSelfDeprecation, bool $doNotFailOnDirectDeprecation, bool $doNotFailOnIndirectDeprecation, bool $doNotFailOnPhpunitDeprecation, bool $doNotFailOnPhpunitNotice, bool $doNotFailOnPhpunitWarning, bool $doNotFailOnEmptyTestSuite, bool $doNotFailOnIncomplete, bool $doNotFailOnNotice, bool $doNotFailOnRisky, bool $doNotFailOnSkipped, bool $doNotFailOnWarning, int $stopOnDefect, int $stopOnDeprecation, ?string $specificDeprecationToStopOn, int $stopOnError, int $stopOnFailure, int $stopOnIncomplete, int $stopOnNotice, int $stopOnRisky, int $stopOnSkipped, int $stopOnWarning, bool $outputToStandardErrorStream, int $columns, bool $noExtensions, ?string $pharExtensionDirectory, array $extensionBootstrappers, bool $backupGlobals, bool $backupStaticProperties, bool $beStrictAboutChangesToGlobalState, bool $colors, bool $processIsolation, bool $enforceTimeLimit, int $defaultTimeLimit, int $diffContext, int $timeoutForSmallTests, int $timeoutForMediumTests, int $timeoutForLargeTests, bool $reportUselessTests, bool $strictCoverage, bool $requireCoverageContribution, bool $disallowTestOutput, bool $displayDetailsOnAllIssues, bool $displayDetailsOnIncompleteTests, bool $displayDetailsOnSkippedTests, bool $displayDetailsOnTestsThatTriggerDeprecations, bool $displayDetailsOnPhpunitDeprecations, bool $displayDetailsOnPhpunitNotices, bool $displayDetailsOnTestsThatTriggerErrors, bool $displayDetailsOnTestsThatTriggerNotices, bool $displayDetailsOnTestsThatTriggerWarnings, bool $reverseDefectList, bool $requireCoverageMetadata, bool $requireSealedMockObjects, bool $noProgress, bool $noResults, bool $noOutput, int $executionOrder, int $executionOrderDefects, bool $resolveDependencies, ?string $logfileTeamcity, ?string $logfileJunit, ?string $logfileOtr, bool $includeGitInformation, bool $includeGitInformationInOtrLogfile, ?string $logfileTestdoxHtml, ?string $logfileTestdoxText, ?string $logEventsText, ?string $logEventsVerboseText, bool $compactOutput, bool $teamCityOutput, bool $testDoxOutput, bool $testDoxOutputSummary, ?array $testsCovering, ?array $testsUsing, ?array $testsRequiringPhpExtension, ?string $filter, ?string $excludeFilter, ?string $testIdFilterFile, ?string $testIdFilter, array $groups, array $excludeGroups, int $randomOrderSeed, int $repeat, int $retry, bool $includeUncoveredFiles, TestSuiteCollection $testSuite, string $includeTestSuite, string $excludeTestSuite, ?string $defaultTestSuite, bool $ignoreTestSelectionInXmlConfiguration, array $testSuffixes, Php $php, bool $controlGarbageCollector, int $numberOfTestsBeforeGarbageCollection, ?string $generateBaseline, bool $debug, bool $withTelemetry, int $shortenArraysForExportThreshold, bool $warnWhenPhpIsNotConfiguredForDevelopment, bool $cacheTestIndex, bool $recordTestImpactData, bool $deriveTestImpactDataFromCoverageTargets)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.49%. Comparing base (f27c338) to head (447d287).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6919      +/-   ##
============================================
+ Coverage     99.48%   99.49%   +0.01%     
- Complexity     9608    10028     +420     
============================================
  Files           922      939      +17     
  Lines         29171    30191    +1020     
============================================
+ Hits          29020    30040    +1020     
  Misses          151      151              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.