Skip to content

SPIKE: Kekiri as a native MTP test runner - #37

Draft
chris-peterson wants to merge 7 commits into
upgrade-xunit-v3-net8from
spike/mtp-native-runner
Draft

SPIKE: Kekiri as a native MTP test runner#37
chris-peterson wants to merge 7 commits into
upgrade-xunit-v3-net8from
spike/mtp-native-runner

Conversation

@chris-peterson

@chris-peterson chris-peterson commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Discovery and execution are Kekiri's own work; they just reach the outside world
through xUnit's or NUnit's extensibility points, which is why Kekiri.Xunit and
Kekiri.NUnit both exist to do the same job twice. This spike asks whether the
host framework is needed at all. Kekiri.Mtp is an ITestFramework on
Microsoft.Testing.Platform, so a test project references it and nothing else: no
xUnit, no NUnit, no VSTest adapter.

What that buys is a scenario's Gherkin shape reaching the runner, the reports and
the IDE as data rather than as console text, which is what makes a BDD-shaped
tree and an RSpec-style report possible. Not for merge: it's a spike, kept out
of Kekiri.slnx so a solution-wide dotnet test stays green.

Depends on #36, which has to merge first (GitHub won't enforce that).

Review guide

Start here, the runner itself

Worth your attention

  • FailureReport.cs L66, which wrappers get unwrapped. GivenFailed/WhenFailed/ThenFailed are noise, but WrongExceptionType and ExpectedExceptionNotCaught are the diagnosis, and unwrapping past those reports the exception the scenario asked for as though it were the failure. StepName is the discriminator
  • Scenario.cs L201, an [Example(2)] against a decimal parameter arrives boxed as int, because reflection does none of the widening a call site would
  • Scenario.cs L126, TestMethodIdentifierProperty has to name a type that exists. Putting the feature here (empty namespace, spaces in the name) cost the entire test tree, and Rider reports that as "no tests" rather than as a bad identifier. Feature grouping rides on the Feature trait instead
  • ScenarioFilter.cs L15, TestNodeUidListFilter is how an IDE asks for one scenario. Ignoring the filter is what makes "run this test" run all of them

Changes outside the spike directory. These live in Kekiri core and affect the
xUnit and NUnit runners' output too, so they need a call on where they belong:

  • ScenarioException.cs L9 adds StepName (which step failed) and Reason (the message without its Error in '<scenario>': header). Both are read-only additions on an internal type
  • ExpectedExceptionNotCaught.cs L8, the exception-expectation messages now open with Expected and use the short type name, so a truncated tree row still says what was wanted. WrongExceptionType already did both

Context, at your leisure

  • README.md, the findings, including what the protocol won't do and what a real version would need
  • GherkinFormatter.cs L28, the RSpec-style output
  • Kekiri.Examples.Mtp.csproj L21, two packages that look like VSTest leftovers and aren't. Microsoft.NET.Test.Sdk is the only thing that sets IsTestProject=true, and Microsoft.Testing.Platform.MSBuild supplies the InvokeTestingPlatform and Test targets. Without them an IDE shows an empty test explorer while the executable runs fine

Approach & trade-offs

Grouping can't be Gherkin-shaped. MTP gives a framework no way to name its own
tree levels. The namespace/type in TestMethodIdentifierProperty is the only lever
and it must name real code; a container TestNode gets counted as a test
(--list-tests reported 5 for 3 scenarios); and ParentTestNodeUid has no
consumer yet. So the tree is namespace/fixture, and Feature is a trait.

dotnet test swallows the RSpec output. MTP substitutes a passthrough output
device there, so the format is available to a direct run and a watch loop, not to
CI through dotnet test.

Validation

Run against a large unit suite
swapping using Kekiri.Xunit for using Kekiri.Mtp and replacing its
[assembly: AssemblyFixture] with a Program.cs that calls
AutofacBootstrapper.Initialize directly:

Evidence Result
Scenarios discovered 2839, matching an independent reflection count of 442 [Scenario] plus 2397 [Example] rows
Suite result 2839 passed, ~14s
Rider tree, per-test run, and Gherkin in the output pane
Not carried plain [Fact]/[Theory] tests, which have no equivalent here; 24 of 95 Kekiri projects have them

That exercise is what surfaced outlines, argument coercion, traits, filters and the
failure-reporting fixes. Four example scenarios found none of them.

@chris-peterson
chris-peterson force-pushed the spike/mtp-native-runner branch from 6dedcd8 to 51de69d Compare August 20, 2026 18:15
@chris-peterson chris-peterson self-assigned this Aug 20, 2026
@chris-peterson
SPIKE: run Kekiri scenarios with no xUnit or NUnit underneath
e2e08bf 
A working ITestFramework on Microsoft.Testing.Platform, to see what a BDD
runner that owns its own output and navigation would look like. Not for
merge, and deliberately outside Kekiri.slnx so a solution-wide dotnet test
stays green -- the example fails one scenario on purpose, because terminal
reporters only print a test's output when it fails, and that output is the
point.

The runner is four files, and the example project references neither xunit
nor NUnit nor Microsoft.NET.Test.Sdk. Discovery and execution were always
Kekiri's work; what owning the TestNode buys is that the Gherkin shape
travels as data rather than as console text. The scenario title becomes the
display name, the Given/When/Then lines ride along as StandardOutputProperty
so a failure shows the scenario instead of only a stack trace, the feature
becomes a trait for grouping, and [CallerFilePath] on [Scenario] gives an IDE
somewhere to navigate without reading a PDB.

Three things the protocol won't do, each found by trying it:

- A container node counts as a test. Publishing a "Feature:" node to group
  scenarios made --list-tests report 5 tests for 3 scenarios, so grouping has
  to be a trait.
- ParentTestNodeUid exists on TestNodeUpdateMessage and the docs describe it
  as enabling future tree-based features, but the terminal reporter flattens
  it today. A real Feature -> Scenario -> Step tree is a capability without a
  consumer.
- DisplayName cannot hold newlines; they render as literal escapes.

Also worth recording: TestMethodIdentifierProperty takes a method-arity
argument that the published signature in the docs omits.

IDE behavior is unverified and is the open question. Everything here was
checked through the terminal reporter and --list-tests; whether Rider, Visual
Studio and VS Code group by the Feature trait or honour the file location
can't be answered from a shell.

The runner reaches the Gherkin text through an InternalsVisibleTo and an
internal IReportTarget, which is the one change outside the spike directory.
Supporting this properly would mean a public contract for "what steps did
this scenario run", plus step-level timing -- TimingProperty already takes
StepTimingInfo[], and the docs call it out for exactly this shape, but Kekiri
times the scenario rather than the steps.
@chris-peterson
SPIKE: emit the run as Gherkin, RSpec documentation style
ddd0d84 
The scenario text becomes the progress report instead of something only a
failure reveals:

    Feature: Addition

      Scenario: Adding 50 and 70
        Given a calculator
          And the user enters 50
          And the user enters 70
        When adding
        Then the result is 120
        passed (2ms)

This works because the platform's own reporter prints nothing per test at
default verbosity, so the Gherkin is the output rather than competing with
it. It goes through IOutputDevice -- the service extensions use to write to
the terminal -- rather than Console.Write, so the platform still owns
ordering and redirection.

Kekiri wraps a step failure in its own exception whose first line names the
scenario, so the summary line walks to the innermost exception; otherwise
every failure reads "Error in 'Adding_two_numbers':" and never says why.

The limitation worth knowing: dotnet test swallows this. MTP substitutes a
passthrough output device there, and --output Detailed doesn't bring it back
(that flag also collides with dotnet test's own --output). So the format is
available to a direct run and a watch loop, not to CI through dotnet test as
it stands. IDEs are a separate question either way, since they read the
protocol's nodes rather than this console stream -- which is why the step
text is also attached as StandardOutputProperty.
@chris-peterson
SPIKE: report the cause, not Kekiri's wrapper around it
2ee8430 
A step failure arrives wrapped twice: ThenFailed names the step, and
ScenarioException prefixes the scenario. Reported as-is, the reader met the
wrapper's message and five frames of Kekiri internals before the arrow that
led to what actually went wrong.

    Error in 'Adding_two_numbers':
    'the result is 120' failed
      at Kekiri.Impl.ScenarioRunner.InvokeThensAsync()
      at Kekiri.Impl.ScenarioRunner.RunAsync()
      at Kekiri.ScenarioBase.RunAsync()
      at Kekiri.ScenarioBase.RunAsync()
      at Kekiri.Mtp.Scenario.RunAsync()
    --->  System.Exception: Expected 120 but got 50

The node now carries the innermost exception, so the first line is the cause
and the top of the stack is the step the reader wrote:

    Expected 120 but got 50
      at ...Adding_two_numbers.the_result_is_120() in Adding_two_numbers.cs:38

Nothing is lost by dropping the wrapper, because the scenario and step names
are what the Gherkin output is already showing -- and it now marks the failing
step in place rather than appending a verdict at the end:

      When adding
    x Then the result is 120
        Expected 120 but got 50

Finding which step failed needed the step name, which existed only inside the
message text. ScenarioException carries it as StepName now, rather than the
runner matching on prose that exists to be read. Both types are internal, so
no public surface changes, and the message strings are untouched -- but this
is a change outside the spike directory and belongs on the migration branch
rather than here if it's kept.
@chris-peterson
SPIKE: colour the Gherkin output
337e48f 
Cyan feature headers, green on a passing verdict, red on a failing step and
its cause. RSpec's colours, so a run is scannable without reading it.

Set through FormattedTextOutputDeviceData.ForegroundColor rather than by
writing ANSI, which leaves the platform deciding when colour is wanted:
piping the run emits no escape codes at all, and --no-ansi and CI are handled
without this code knowing about either. The cost is that checking it needs a
terminal, since a captured run is colourless by design -- script -q /dev/null
against the executable shows the codes.

The blank separator lines are written separately from the coloured text. With
the newline inside the coloured span the escape landed before it, so a colour
code sat on an empty line; it rendered correctly but read badly in a captured
run.
@chris-peterson
Target net10.0 in the Autofac xUnit example
88bb318 
Matches the MTP example alongside it. The shipped packages stay on net8.0;
only this example project moves, and a net10.0 consumer of net8.0 packages is
the normal direction. CI installs both runtimes already.
@chris-peterson
SPIKE: show the scenarios in Rider's test tree
a7423eb 
Rider ran the executable and rendered its output, but listed no tests for the
project, which makes the whole approach a non-starter however good the console
output is.

Microsoft.NET.Test.Sdk is what was missing, and it is the only thing in the
graph that sets IsTestProject=true. The Testing Platform capabilities Rider
talks over -- TestingPlatformServer and TestContainer, declared by
Microsoft.Testing.Platform's own targets -- already arrived through the
ProjectReference to Kekiri.Mtp, so those were never the gap; a two-project probe
confirms IsTestingPlatformApplication is true without the package. JetBrains
names the same package as the workaround in RIDER-129745. Referencing it means
GenerateProgramFile=false, since Program.cs writes its own Main.

With a tree to look at, three things it showed were worth fixing:

The tree grouped by the full namespace path and then the declaring fixture.
Gherkin has two levels, so the feature now goes into
TestMethodIdentifierProperty as the type with an empty namespace, and the tree
reads Addition -> Adding 50 and 70. A fixture is a C# artifact rather than part
of the spec, and one feature's scenarios are usually spread over several of them.
Identity still travels in Uid and navigation in TestFileLocationProperty, so
neither depends on that property naming a real type.

Names arrived as identifiers. Names.Sentence renders both conventions a suite
mixes -- Adding_two_numbers and AddingTwoNumbers both read "Adding two numbers"
-- and leaves an all-capitals run alone so HTTP stays HTTP.

A failure carried the frames Kekiri used to reach the step method underneath the
step itself. StackTraceFilter drops those, as xUnit and NUnit both do. It has to
hand the reporter a substitute exception overriding StackTrace, because
FailedTestNodeStateProperty takes the trace off the Exception it carries.

Which field Rider reads the grouping from is still untested: the Uid spells out
the same type, so only one of the two has to be right.
@chris-peterson
chris-peterson force-pushed the spike/mtp-native-runner branch from 51de69d to 13951c2 Compare August 21, 2026 13:58
@chris-peterson
Rename to Behavior and ship it as one package
68fe204 
Pointing the runner at a 2839-scenario suite found the gaps four example
scenarios never could, and the shape that came out of fixing them is a
product rather than a spike. This commit is that: what the real suite
demanded, and the name and packaging it gets as a result.

Kekiri's runners lived inside a host test framework, so the packages were
split along that seam: an engine, plus one runner package per framework.
With discovery and execution owned outright there is one runner, and the
split has nothing left to express -- the IoC packages are the only other
consumers of the engine, and they are used exclusively from inside a
scenario project that already carries it. Engine and runner fold into a
single `Behavior` package, with `Behavior.Autofac` and
`Behavior.ServiceProvider` beside it, all at 0.1.0.

The new name comes with new package ids on purpose. `Kekiri`,
`Kekiri.Xunit`, `Kekiri.NUnit` and the `Kekiri.IoC.*` packages stay on
NuGet at the versions they are today, so an existing suite cannot be
moved onto this by an update -- migrating is an explicit act, and
docs/migrating.md is what it takes. The repository keeps its name, so
bookmarks, the docs site path and the wiki all still resolve.

What the suite run demanded, in the order it surfaced:

Example rows. 2397 of those 2839 tests are [Example] rows, so the runner
reached almost none of the suite without them. There is no separate
outline attribute: xUnit needed Fact and Theory to be different things
and dispatched discovery per attribute, and neither constraint survives
owning discovery, so [Scenario] is the only marker and its [Example] rows
decide whether it expands. Two things the naive version got wrong: an
attribute argument keeps the type its literal had, so an [Example(2)]
against a decimal parameter arrives boxed as int and MethodInfo.Invoke
rejects it -- reflection does none of the widening a call site would; and
ordering rows by their title alphabetized them, so the tree disagreed
with the Examples table.

Reaching the recorded Gherkin was reflecting for a non-public field,
which finds nothing when it is declared on a base class rather than the
test's own type. The example suite never noticed because it subclasses
Scenarios directly; a real one goes four deep. It is an interface now.

Tags, because the suite tags scenarios with a spec reference and Gherkin
tags are the same idea. [Tag(name, value)] carries any pair and
[Category(value)] is the shorthand every .NET runner already spells that
way; both become TestMetadataProperty. The feature joins them in
FilterProperties, so a filter can name the grouping the reader sees
rather than only the tags.

Rider needed two corrections. Feature-as-type grouping is out:
TestMethodIdentifierProperty has to name a type that exists, and
substituting a feature (empty namespace, spaces in the name) costs the
whole tree, which an IDE reports as no tests rather than as a bad
identifier. Feature grouping rides on metadata instead, which is a value
rather than an identifier. And the filter on each request was ignored,
which is what made "run this one test" run all of them:
TestNodeUidListFilter is how an IDE asks for one scenario.
--treenode-filter is handled too, and needed registering.

Failure reporting told the wrong story for an expected exception.
Reporting the innermost exception is right for the step wrappers, whose
whole message is "'<step>' failed", and wrong for the
exception-expectation wrappers, which are themselves the diagnosis and
hold the cause as their inner. Unwrapping past WrongExceptionType or
ExpectedExceptionNotCaught reports the exception the scenario asked for
as though it were the failure, so a test reads as failing for the reason
it was written to expect. Only the step wrappers carry a StepName, so
that is the discriminator rather than a list of type names.

The remaining case had no message at all: ScenarioException.Message opens
with an "Error in '<scenario>':" header and puts the cause on the second
line, so taking the first line yielded a header naming a scenario already
on screen. The class now exposes the bare Reason, since it knows both
halves and splitting the string back apart would be guesswork. Also
dropped from a trace: the runtime's InvokeStub_ frames, named after the
type they were emitted for so they share no namespace with the
System.Reflection entries already filtered; and the trace entirely when
every frame is machinery, which is the truth for a compliance failure
detected after the last step rather than inside one.

The messages themselves are shortened to fit a tree row, and to read as
one family: every one now opens with "Expected", so a truncated row still
says what was wanted. They use the short type name, which
WrongExceptionType already did -- the fully qualified name in the other
two was the odd one out, and it cost 60 characters before the sentence
reached its verb. Catch<T> keeps its type argument, so the message names
the call that is missing rather than describing it.

The rename broke one thing nothing covered: the frame filter dropped
stack frames whose namespace started with the framework's own root, and
that root is now a prefix a consumer might plausibly write -- every frame
the reader wrote would have been stripped from a failure. The filter
names the three internal namespaces instead, and a TUnit suite covers
that along with naming, discovery, tags, filtering and execution.
@chris-peterson
chris-peterson force-pushed the spike/mtp-native-runner branch from 13951c2 to 68fe204 Compare August 24, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@chris-peterson chris-peterson

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chris-peterson