Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix - #54231
Closed
MichaelSimons wants to merge 4 commits into
Closed
Convert dotnet-format integration tests from pipeline YAML to XUnit/Helix#54231MichaelSimons wants to merge 4 commits into
MichaelSimons wants to merge 4 commits into
Conversation
Member
Author
|
@mwiemer-microsoft, @phil-allen-msft - I had copilot convert the dotnet-format integration tests to helix in order to save compute as well as make the tests just list any other test. As part of this work, we only run a subset of the tests in PR (as discussed with Phil earlier). This is WIP but wanted to share to get your early feedback. |
This was referenced May 11, 2026
MichaelSimons
force-pushed
the
format-timings
branch
6 times, most recently
from
May 21, 2026 19:41
a5268f7 to
dee8c77
Compare
MichaelSimons
force-pushed
the
format-timings
branch
2 times, most recently
from
June 5, 2026 18:59
4be8a41 to
dc54845
Compare
Member
Author
|
/ba-g all test failures appear related to #54608 |
MichaelSimons
force-pushed
the
format-timings
branch
3 times, most recently
from
June 30, 2026 20:55
772b70a to
0fba789
Compare
…elix - Migrate test project from xUnit to MSTest (MSTest.Sdk) - Remove AspNetCore and EfCore test scenarios (restore exceeds Helix timeout) - Refactor RunProcess with better parameter defaults (captureOutput=false, ignoreThrowingOnError=false) - Remove assembly-level parallelization attribute (repo-wide setting) - Use TestContextOutputHelper for test output instead of ITestOutputHelper constructor injection - Replace IAsyncLifetime with [TestInitialize] lifecycle Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Simplify Restore() to always use 'dotnet restore <solution>' with the SDK under test - Move RemoveGlobalJson() before Restore() so our SDK is used for both restore and format - Remove RunBuildScriptRestore() (Arcade restore didn't generate project.assets.json) - Remove unused UseRepoBuildScript property and cleanSdkEnvironment parameter - Fix loop variable mutation bug in SDKCustomCreateXUnitWorkItemsWithTestExclusion - Hoist enableDiagLogging computation outside the loop Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Parse 'Formatted X of Y files' from dotnet format output (requires --verbosity detailed) and assert Y > 0. This catches false-positive passes where a broken restore causes format to load zero projects and exit 0 silently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Instead of deleting global.json entirely, strip only the 'sdk' section. This preserves the 'msbuild-sdks' section that tells NuGet SDK resolver which versions of MSBuild SDKs (e.g., Microsoft.DotNet.Arcade.Sdk, Microsoft.Build.NoTargets, Microsoft.Build.Traversal) to resolve. Without this, repos like Roslyn and MSBuild fail to restore because their project files import Arcade targets that depend on the SDK version being pinned in global.json's msbuild-sdks section. Also adds RunArcadeBuildScriptRestore() which runs the repo's eng/Build.ps1 (or eng/build.sh) -restore to install Arcade tooling before global.json modification, matching the original format-verifier.ps1 behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Converts the dotnet-format integration test pipeline legs (YAML + PowerShell) into MSTest integration tests that run on Helix. This eliminates redundant SDK builds and reduces PR validation cost.
Time Savings
Measured from build 1412916:
AspNetCore and EfCore dropped: These repos are restore-intensive and not feasible to run on Helix. The ROI is questionable for a format validation check.
What changed
test/dotnet-format.IntegrationTests/using MSTest.Sdk with one test class per repo (SDK, Roslyn, MSBuild, ProjectSystem)FormatIntegrationTestBasehandles clone, restore, and format execution. Adding or removing a repo requires only adding or deleting a ~15-line class.PartitionByClass=truemetadata causesAssemblySchedulerto discover test classes at scheduling time and create one Helix work item per class automatically.DOTNET_SDK_TEST_MAX_PRIORITYenv var (set in the pipeline YAML for PR builds) and a numericPriorityproperty control which tests run in each context. P0 tests run in all builds (including PR validation); P1 tests (the default) run only in CI.eng/dotnet-format/(YAML template, PowerShell verifier, CMD wrapper, RSP file) and removed pipeline references from.vsts-ci.yml/.vsts-pr.yml.Design
Each repo is a separate test class inheriting from
FormatIntegrationTestBase:This one-class-per-repo pattern enables:
DOTNET_SDK_TEST_MAX_PRIORITY=0for PR builds (unset in CI = run all priorities). Adding a new priority tier requires no test code changes, only a new env var value in the pipeline.