Skip to content

Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix - #54231

Closed
MichaelSimons wants to merge 4 commits into
dotnet:mainfrom
MichaelSimons:format-timings
Closed

Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231
MichaelSimons wants to merge 4 commits into
dotnet:mainfrom
MichaelSimons:format-timings

Conversation

@MichaelSimons

@MichaelSimons MichaelSimons commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

Converts the dotnet-format integration test pipeline legs (YAML + PowerShell) into MSTest integration tests that run on Helix. This eliminates redundant SDK builds and reduces PR validation cost.

Time Savings

Measured from build 1412916:

Leg Old Leg Time PR CI
sdk 19m 03s ✅ Runs (saves ~8m build) ✅ Runs (saves ~8m build)
Roslyn 28m 50s ⏭️ Skipped (saves ~29m) ✅ Runs (saves ~8m build)
msbuild 17m 02s ⏭️ Skipped (saves ~17m) ✅ Runs (saves ~8m build)
project-system 12m 31s ⏭️ Skipped (saves ~13m) ✅ Runs (saves ~9m build)
efcore 15m 34s ❌ Dropped (saves ~16m) ❌ Dropped (saves ~16m)
aspnetcore 25m 34s ❌ Dropped (saves ~26m) ❌ Dropped (saves ~26m)
Total savings ~101m ~75m

AspNetCore and EfCore dropped: These repos are restore-intensive and not feasible to run on Helix. The ROI is questionable for a format validation check.

What changed

  • New test project: test/dotnet-format.IntegrationTests/ using MSTest.Sdk with one test class per repo (SDK, Roslyn, MSBuild, ProjectSystem)
  • Base class pattern: FormatIntegrationTestBase handles clone, restore, and format execution. Adding or removing a repo requires only adding or deleting a ~15-line class.
  • Dynamic Helix sharding: PartitionByClass=true metadata causes AssemblyScheduler to discover test classes at scheduling time and create one Helix work item per class automatically.
  • Priority-based test filtering: DOTNET_SDK_TEST_MAX_PRIORITY env var (set in the pipeline YAML for PR builds) and a numeric Priority property control which tests run in each context. P0 tests run in all builds (including PR validation); P1 tests (the default) run only in CI.
  • Removed old infrastructure: Deleted eng/dotnet-format/ (YAML template, PowerShell verifier, CMD wrapper, RSP file) and removed pipeline references from .vsts-ci.yml / .vsts-pr.yml.

Design

Each repo is a separate test class inheriting from FormatIntegrationTestBase:

[TestClass]
public class SdkFormatTests : FormatIntegrationTestBase
{
    protected override string RepoUrl => "https://github.com/dotnet/sdk";
    protected override string Sha => "e6bc966cc3d1348265b0831c6daca23267169d8f";
    protected override string TargetSolution => "sdk.slnx";
    protected override int Priority => 0; // Runs in PR validation
}

This one-class-per-repo pattern enables:

  1. Helix sharding — each class becomes its own work item (parallel execution)
  2. Zero maintenance — adding or removing a repo only requires adding or deleting a class file; sharding adapts automatically
  3. Extensible priority filtering — any test class can declare its priority level. The CI infrastructure sets DOTNET_SDK_TEST_MAX_PRIORITY=0 for PR builds (unset in CI = run all priorities). Adding a new priority tier requires no test code changes, only a new env var value in the pipeline.

@MichaelSimons MichaelSimons changed the title Add timing instrumentation to dotnet-format integration tests Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix May 8, 2026
@MichaelSimons

Copy link
Copy Markdown
Member Author

@mwiemer-microsoft, @phil-allen-msft - I had copilot convert the dotnet-format integration tests to helix in order to save compute as well as make the tests just list any other test. As part of this work, we only run a subset of the tests in PR (as discussed with Phil earlier). This is WIP but wanted to share to get your early feedback.

@MichaelSimons

Copy link
Copy Markdown
Member Author

/ba-g all test failures appear related to #54608

@MichaelSimons
MichaelSimons force-pushed the format-timings branch 3 times, most recently from 772b70a to 0fba789 Compare June 30, 2026 20:55
MichaelSimons and others added 4 commits June 30, 2026 15:55
…elix

- Migrate test project from xUnit to MSTest (MSTest.Sdk)
- Remove AspNetCore and EfCore test scenarios (restore exceeds Helix timeout)
- Refactor RunProcess with better parameter defaults (captureOutput=false, ignoreThrowingOnError=false)
- Remove assembly-level parallelization attribute (repo-wide setting)
- Use TestContextOutputHelper for test output instead of ITestOutputHelper constructor injection
- Replace IAsyncLifetime with [TestInitialize] lifecycle

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Simplify Restore() to always use 'dotnet restore <solution>' with the SDK under test
- Move RemoveGlobalJson() before Restore() so our SDK is used for both restore and format
- Remove RunBuildScriptRestore() (Arcade restore didn't generate project.assets.json)
- Remove unused UseRepoBuildScript property and cleanSdkEnvironment parameter
- Fix loop variable mutation bug in SDKCustomCreateXUnitWorkItemsWithTestExclusion
- Hoist enableDiagLogging computation outside the loop

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parse 'Formatted X of Y files' from dotnet format output (requires
--verbosity detailed) and assert Y > 0. This catches false-positive
passes where a broken restore causes format to load zero projects
and exit 0 silently.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Instead of deleting global.json entirely, strip only the 'sdk' section.
This preserves the 'msbuild-sdks' section that tells NuGet SDK resolver
which versions of MSBuild SDKs (e.g., Microsoft.DotNet.Arcade.Sdk,
Microsoft.Build.NoTargets, Microsoft.Build.Traversal) to resolve.

Without this, repos like Roslyn and MSBuild fail to restore because
their project files import Arcade targets that depend on the SDK version
being pinned in global.json's msbuild-sdks section.

Also adds RunArcadeBuildScriptRestore() which runs the repo's eng/Build.ps1
(or eng/build.sh) -restore to install Arcade tooling before global.json
modification, matching the original format-verifier.ps1 behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants