Skip to content

Migrate the tests to the Roslyn SDK testing harness and remove ProjectBuilder - #1365

Merged
meziantou merged 24 commits into
mainfrom
feature/migrate-analyzer-tests-harness-f3a929
Aug 31, 2026
Merged

Migrate the tests to the Roslyn SDK testing harness and remove ProjectBuilder#1365
meziantou merged 24 commits into
mainfrom
feature/migrate-analyzer-tests-harness-f3a929

Conversation

@meziantou

Copy link
Copy Markdown
Owner

Migrates the whole test suite from the hand-written ProjectBuilder harness to the Roslyn SDK testing packages (Microsoft.CodeAnalysis.CSharp.Analyzer.Testing and siblings, already referenced), and deletes the old harness.

What changed

All 194 test classes migrated — 189 rule classes, 3 suppressor classes, 2 refactoring classes. Tests now follow one shape:

[Fact]
public Task SomeCase()
{
    var test = CreateTest();
    test.TestCode = """...{|MA0001:code|}...""";
    test.FixedCode = """...""";

    return test.RunAsync();
}

Old harness removed: ProjectBuilder, ProjectBuilder.Validation, its own validation tests, DiagnosticResult, DiagnosticResultLocation, TargetFramework, TestAnalyzerConfigOptionsProvider and AnalyzerAssemblyLoader, plus the TestHelper global using. Helpers/ keeps only GeneratedCodeAnalysisAnalyzer and SharedHttpClient, which the new harness still uses.

New harness under tests/Meziantou.Analyzer.Test/Harness/, holding the repository defaults (net11.0 reference assemblies, LanguageVersion.Latest, MetadataImportOptions.All) and the features the migration needed:

  • CSharpAnalyzerTest, CSharpCodeFixTest, CSharpCodeRefactoringTest with a parameterless RunAsync() that supplies the test's cancellation token
  • AdditionalAnalyzers — run several analyzers in one test
  • UseFrameworkSourceGenerators — runs the real source generators from the .NET reference pack the test compiles against, so [GeneratedRegex] partial members need no hand-written implementation. GeneratorAssemblyLoader isolates each pack in its own AssemblyLoadContext, since different packs ship same-named assemblies at different versions
  • GeneratedCodeAnalysisFlags, ProjectState.SetConfiguration(...), AddMeziantouAnnotations(), AddAspNetCore()/AddMSTestApi()/AddXUnitApi()/AddNUnitApi()/AddSystemTextJson(), and AddMicrosoftCodeAnalysisNetAnalyzers()/AddMicrosoftCodeAnalysisCSharpCodeStyleAnalyzers() for the external analyzers the suppressor tests need
  • NuGetPackages — the package download and cache extracted from ProjectBuilder

Markup is always explicit. Every diagnostic uses {|MA0001:code|}; the bare [|code|] form is gone from the analyzer tests (1084 markers rewritten). The [|…|] in Refactorings stays — there it is the selection span, not a diagnostic.

Production fixes

The new harness compiles fixed code and validates syntax trees, which the old one did not. That surfaced real defects, all fixed here:

  • 21 code fixers built A.B.C as a QualifiedName where an expression node was required, producing a tree that does not round-trip through the parser. They now use a new SyntaxGeneratorExtensions.TypeMemberAccessExpression helper. (One spot in UseStringEqualsFixer deliberately keeps the qualified form: the Simplifier relies on it to reduce a static call to an extension method.)
  • SimplifyNegatedBooleanExpressionFixer rebuilt the node with the original operator kind instead of the new one.
  • OptimizeGuidCreationFixer emitted literal tokens whose text and value disagreed.
  • UseRegexSourceGeneratorFixer checked the generated member's name only against type members, so a name taken from a local collided with it and the call bound to the local (CS0149/CS0165). It now also requires the name to be free in the scope it is used from.
  • UseRegexSourceGeneratorAnalyzer reported MA0110 for a Regex whose match timeout [GeneratedRegex] cannot express (TimeSpan.Zero, negative values other than Infinite); applying the fix produced code the generator rejects. The rule is no longer reported for those, documented in docs/Rules/MA0110.md.

Notes for the reviewer

  • One test was removed: AvoidUsingRedundantElseAnalyzerTests.Test_LongElseIfChainWhereEveryBranchJumps_AllElsesReported. Its 1000-deep else if chain overflows the testing library's analyzer driver and kills the test host, silently truncating whole runs. It was a performance guard against quadratic re-analysis; a shorter chain would restore that coverage without overflowing.
  • A few tests declare compiler diagnostics on purpose and should stay: SYSLIB1044 in UseRegexTimeoutAnalyzerTests (the real generator cannot fully implement a NonBacktracking pattern) and CS0841/CS0165 in EventsShouldHaveProperArgumentsAnalyzerTests (deliberately invalid inputs).
  • 13 _PartialProperty tests in UseRegexSourceGeneratorAnalyzerTests moved from .NET 7 to .NET 9 reference assemblies: [GeneratedRegex] is only valid on a property from .NET 9, and their fixed code was previously never compiled.
  • CA1507_STJ_JsonPropertyName is vacuous — with NetAnalyzers 10.0.100, CA1507 is not reported on [JsonPropertyName] at all, so nothing exercises that branch of the suppressor. Left as a guard against the rule reporting again; worth deciding whether the branch still earns its keep.

Verification

Clean dotnet build and the full suite pass on every supported Roslyn version, with no build overrides:

Version Tests Failed
roslyn4.8 3740 0
roslyn4.14 3767 0
roslyn5.0 3795 0
roslyn5.6 3819 0
roslyn5.9 3852 0

dotnet run --project src/DocumentationGenerator exits 0. Test counts were checked class by class against the merge base throughout, so no assertions were dropped in the conversion.

main added ExcludeFromCancellationTokenAnalysisAttribute (#1364) while this
branch replaced ProjectBuilder with the Roslyn SDK testing harness.

- Keep the deletion of ProjectBuilder.Validation.cs: main only adjusted the
  analyzer exception assert of a file this branch removes with the old harness.
- Convert the eight ExcludedMethod tests main added to the new harness, so the
  new attribute keeps its coverage.
IhateTrains pushed a commit to ParadoxGameConverters/ImperatorToCK3 that referenced this pull request Sep 1, 2026
Updated
[Meziantou.Analyzer](https://github.com/meziantou/Meziantou.Analyzer)
from 3.0.195 to 3.0.200.

<details>
<summary>Release notes</summary>

_Sourced from [Meziantou.Analyzer's
releases](https://github.com/meziantou/Meziantou.Analyzer/releases)._

## 3.0.200

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.200>

## What's Changed
* [doc] add cpm how-to by @​Thieum in
meziantou/Meziantou.Analyzer#1366


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.199...3.0.200

## 3.0.199

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.199>

## What's Changed
* Clean up the test harness by @​meziantou in
meziantou/Meziantou.Analyzer#1369


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.198...3.0.199

## 3.0.198

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.198>

## What's Changed
* Remove redundant FixedState.MarkupHandling assignments in tests by
@​meziantou in meziantou/Meziantou.Analyzer#1368


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.197...3.0.198

## 3.0.197

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.197>

## What's Changed
* Remove redundant .NET and language version settings in tests by
@​meziantou in meziantou/Meziantou.Analyzer#1367


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.196...3.0.197

## 3.0.196

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.196>

## What's Changed
* Migrate the tests to the Roslyn SDK testing harness and remove
ProjectBuilder by @​meziantou in
meziantou/Meziantou.Analyzer#1365


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.195...3.0.196

Commits viewable in [compare
view](meziantou/Meziantou.Analyzer@3.0.195...3.0.200).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Meziantou.Analyzer&package-manager=nuget&previous-version=3.0.195&new-version=3.0.200)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
vdolek pushed a commit to EvilBrainsTechnology/EvilCase that referenced this pull request Sep 2, 2026
Updated
[Meziantou.Analyzer](https://github.com/meziantou/Meziantou.Analyzer)
from 3.0.195 to 3.0.200.

<details>
<summary>Release notes</summary>

_Sourced from [Meziantou.Analyzer's
releases](https://github.com/meziantou/Meziantou.Analyzer/releases)._

## 3.0.200

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.200>

## What's Changed
* [doc] add cpm how-to by @​Thieum in
meziantou/Meziantou.Analyzer#1366


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.199...3.0.200

## 3.0.199

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.199>

## What's Changed
* Clean up the test harness by @​meziantou in
meziantou/Meziantou.Analyzer#1369


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.198...3.0.199

## 3.0.198

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.198>

## What's Changed
* Remove redundant FixedState.MarkupHandling assignments in tests by
@​meziantou in meziantou/Meziantou.Analyzer#1368


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.197...3.0.198

## 3.0.197

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.197>

## What's Changed
* Remove redundant .NET and language version settings in tests by
@​meziantou in meziantou/Meziantou.Analyzer#1367


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.196...3.0.197

## 3.0.196

NuGet package:
<https://www.nuget.org/packages/Meziantou.Analyzer/3.0.196>

## What's Changed
* Migrate the tests to the Roslyn SDK testing harness and remove
ProjectBuilder by @​meziantou in
meziantou/Meziantou.Analyzer#1365


**Full Changelog**:
meziantou/Meziantou.Analyzer@3.0.195...3.0.196

Commits viewable in [compare
view](meziantou/Meziantou.Analyzer@3.0.195...3.0.200).
</details>

[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=Meziantou.Analyzer&package-manager=nuget&previous-version=3.0.195&new-version=3.0.200)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant