Skip to content

Run every registered code action, not just the applied one - #1353

Merged
meziantou merged 4 commits into
mainfrom
test/execute-configured-code-fixers
Aug 27, 2026
Merged

Run every registered code action, not just the applied one#1353
meziantou merged 4 commits into
mainfrom
test/execute-configured-code-fixers

Conversation

@meziantou

@meziantou meziantou commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Stacked on #1354, which fixes the first bug this change found. Review that one first; this PR is red without it.

Problem

Two gaps meant most code fix providers were never really exercised.

1. A provider configured without an expected fixed code was never invoked at all. ValidateAsync only calls VerifyFix when ExpectedFixedCode is not null, so .WithCodeFixProvider<T>() is a no-op otherwise. 355 of the 989 test methods that expect a diagnostic configure a provider that never runs:

test class methods
DoNotUseBlockingCallInAsyncContextAnalyzer_AsyncContextTests 64
DoNotUseImplicitCultureSensitiveToStringAnalyzerTests 42 (2 fix assertions in 88 tests)
AvoidUnusedInternalTypesAnalyzerTests 36
UseIFormatProviderAnalyzerTests 20
UseStringComparerAnalyzerTests 18
NamedParameterAnalyzerTests 16

2. Only one action per diagnostic was ever run. A provider commonly registers several actions; VerifyFix applies actions[codeFixIndex ?? 0] and ignores the rest, so the siblings were unexercised even in tests that do assert a fixed code. And registering a fix does not run it — the delegate passed to CodeAction.Create, where the fix actually builds the new document, only runs when its operations are computed. That delegate is where the casts, the argument indexing and the rewriting live.

Change

  • When no expected fixed code is declared, register the fixes for every reported diagnostic the provider declares as fixable.
  • In both paths, compute the operations of every registered action, which runs each delegate, and let an exception fail the test.

The changes are still not applied when the test declares no expected fixed code — there is nothing to compare them against. Diagnostics are matched to their document by syntax tree, so a diagnostic from an API-reference document is not handed to a provider along with the wrong document. VerifyDiagnostics now returns the documents and diagnostics it already computed so nothing is recomputed. GetSortedDiagnostics had one caller and is inlined into it.

It found a real bug immediately

The first run surfaced an InvalidCastException in OptimizeStringBuilderUsageFixer.RemoveToString for sb.Insert(0, 10.ToString()) — the fixer reads Arguments[0], which is the index for Insert, not the value. Fixed in #1354, which this is stacked on.

Note the earlier version of this PR, which only called RegisterCodeFixesAsync, would not have caught it: the crash is in the action body.

One test had to change

Test_LongElseIfChainWhereEveryBranchJumps_AllElsesReported (added in #1352) now builds an analyzer-only ProjectBuilder. An else if chain is nested syntax, so its 1000 branches are a 1000-level deep tree, and the formatter Roslyn runs while a code action computes its changes recurses once per level and throws InsufficientExecutionStackException. That is a Roslyn limit on deeply nested syntax, not a defect in the fixer, and the test exists to guard the analyzer's complexity.

What this does and does not catch

Catches a provider that throws on a shape no fix assertion covers — unguarded casts, argument-index arithmetic, Single() on an empty sequence.

Does not catch a provider that produces wrong-but-compiling output; that still needs a real ShouldFixCodeWith. The MA0090 bug in #1351 was of that second kind and would not have been caught here.

Cost

Full roslyn5.9 suite: 59.9s on the base branch, 1m02s with this change — about +3s (5%).

All five versions pass: 3708 (4.8) / 3756 (4.14) / 3783 (5.0) / 3807 (5.6) / 3840 (5.9), zero failures.

Not in this PR

Two related harness gaps, left out to keep this reviewable:

  • ValidateAsync's opening guards are dead code — DiagnosticAnalyzer and ExpectedDiagnosticResults are IList<> initialized to [], so is null is never true and a test that forgets .WithAnalyzer<T>() passes silently.
  • ParseSourceCode drops a malformed [|…|] marker without warning, turning a positive test into a vacuous negative one.

A stricter follow-up is also possible: assert that a registered action actually changes the document, which would enforce the AGENTS.md rule against registering a fix that returns the document unchanged. That one will surface real failures, so it is worth doing separately.

The "Remove the ToString call" branch did not exclude Insert, unlike the
string.Format and string.Join branches next to it, so
sb.Insert(0, value.ToString()) was reported. The fix then threw
InvalidCastException: it reads Arguments[0], which is the index for
Insert, not the value, and the analyzer had looked at Arguments[1].

Fixing the argument index alone would not be enough. The fix builds an
Append call, so it would turn an Insert into an Append and drop the
index, and the types the analyzer accepts are the ones that have an
Append overload: StringBuilder and ReadOnlyMemory<char> have no Insert
counterpart, so removing the ToString call would not compile.

Exclude Insert, as the two sibling branches already do. Supporting it
would need its own set of overload types and its own fix.
VerifyFix only runs when a test calls ShouldFixCodeWith or
ShouldBatchFixCodeWith, so WithCodeFixProvider is a no-op for every other
test. 355 of the 989 test methods that expect a diagnostic configure a
provider that is never invoked, even though those snippets are inputs the
provider does run on in an IDE.

When no expected fixed code is declared, call RegisterCodeFixesAsync on
each reported diagnostic the provider declares as fixable, and let an
exception fail the test. The registered actions are not applied, since
the test did not say what the fixed code should be.

VerifyDiagnostics now returns the documents and diagnostics it already
computed so the registration reuses them instead of recreating the
project, which keeps the cost of the suite unchanged (1m00s before and
after, roslyn5.9).

No existing test fails, on any of the five Roslyn versions: this is a
regression guard rather than a fix for a current crash.
Registering the fixes only runs RegisterCodeFixesAsync; the delegate a
provider registers, where the fix actually builds the new document, is
never invoked. And a provider often registers several actions, of which
VerifyFix applies exactly one, so the others were never exercised either.

Compute the operations of every registered action, in both paths, and let
an exception fail the test. The changes are still not applied when the
test declares no expected fixed code, as there is nothing to compare
them to.
@meziantou
meziantou force-pushed the test/execute-configured-code-fixers branch from f0497c4 to f6ac4b6 Compare August 27, 2026 03:34
@meziantou
meziantou changed the base branch from main to fix/ma0028-tostring-on-insert August 27, 2026 03:34
@meziantou meziantou changed the title Run the configured code fix provider even without an expected fixed code Run every registered code action, not just the applied one Aug 27, 2026
Base automatically changed from fix/ma0028-tostring-on-insert to main August 27, 2026 03:35
@meziantou
meziantou merged commit b5df6c5 into main Aug 27, 2026
13 checks passed
@meziantou
meziantou deleted the test/execute-configured-code-fixers branch August 27, 2026 03:48
This was referenced Aug 27, 2026
This was referenced Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant