Proposal: Adding a built-in test runner

[jasnell]

At the risk of opening a whole can of worms given that literally everyone gets super opinionated about test runners... I'd like to propose that we add a built-in test runner to Node.js.

Specifically allowing for something like node --test foo/* to run all tests found in the foo directory ... or node --test foo.js to run all tests found in the foo.js file, etc.

Obviously, this begs the question: Which test runner do we go with. There are options.

1. Vendor in an existing test runner (in which case which should we use?) ... Note: this is not an invitation to start advocating for your specific favorite test runner in this thread. At this point we just need to decide if vendoring in an existing runner is the right choice. We can bike shed on exactly which one that should be later.

2. Implement our own minimalistic test runner with the specific goal of it being extremely small and intentionally lite on features.

[Jamesernator]

I think the most important thing is the ability to scaffold on whatever Node provides that would allow test runners with more features to easily build on top of so that more powerful tests could be enabled.

As one example, suppose someone is wanting to do UI testing with playwright or similar, then they probably want to have certain things like pages made available to each test. Some way of adding things to tests would be neccessary for such things to be ergonomic.

I feel like what would be healthy for the community is for Node to specify a test format that can be scaffolded on top of and provide a basic default implementation on top of that. By having a community standard test format different testing tools could accept the same test files while providing their own features such as optimizations, extra assertions, improved debugging, browser integration, etc etc.

[Trott]

@nodejs/testing

[thebergamo]

I totally agree with @Jamesernator in the regard of creating some base tooling for those test runners be built on top and provide other more advanced features in the user land.

If we look in some of the "modern" programming languages like Rust and Go they already have something built in their std library that can provide out of the box such functionality (limited, but still powerful)

For long time we have had many important and dominant libraries being created as NPM modules so users would be able to choose their own flavors, but if Node.js itself could provide some interfaces/standards itself for those runners be built on top the interoperability between them would be improved significantly and the configuration hassle would be decreased also when thinking in simple or smaller cases for experimentation.

From my user perspective shipping an existing test runner would end up creating other issues for later on in case this runner goes away or a newer and modern one comes out and some requests to "adopt" it in the core could be avoided.

Having a clear state and for sure using those existing as a base for create this abstration would be wider accepted and even adopted for the existing test runners.

[vdeturckheim]

@jasnell does it have to be a cli option or could it be a core lib

// test/foo.js
import { it, describe } from '@node/test';

and call node test/foo.js ? I guess there must be a flag to allow running multiple files in a directory now I write it 🤔

[targos]

@vdeturckheim It probably needs to be both (a flag to enable certain behaviors like watching for changes or coverage output and a core lib for test utilities).

[jasnell]

I would say combination of CLI flags with API, yes.

Specifically:

• CLI for indicating that we want to specifically run tests in a file. It doesn't really need to run all tests in a directory. That's something userland can add. Just something like node --test foo.js == run all tests found in foo.js
• CLI for configuring how to output test results.
• API for declaring tests with support for:
  • Individual tests e.g. test(() => { ... }, 'it works!')
  • Groups of related tests
  • Setup and teardown per test and per group

We already have the built in assert module to use with it. And third parties can extend from there.

[bengl]

There's certainly room for improving support for testing. A limited subset of the functionality provided by all/most test frameworks makes sense to have in Node.js core.

Some further thoughts, some echoing what others have already said in this thread:

• Vendoring an existing test library is a favourite-picking exercise that's going to be fraught with pain, due to rather diverse state of Node.js testing today.
• As previously described here, a minimal runner command (e.g. node --run-tests test/**/*.js) could be used to run test files and determine their pass/fail status based on exit code.
  • This could look very similar to the Node.js core test suite.
  • TAP output is mostly ubiquitous, and so should be the native output here. Userland tools exist for making this more palatable to end-users. (e.g. node --run-tests test/**/*.js | npx tap-colorize)
  • It could also detect TAP output from individual tests and output them as subtests.
  • Options could be added for parallelism, etc.
  • The test files glob could have a default.
• An assertion API is already included. It would be preferable to extend it rather than replace it.
• Agreeing on an API to organize tests might not be easy, but it might be easier to provide some helper/building-block functions to build a test framework out of. For example:
  • A "single test runner" that runs a function, creating a pass/fail status based on whether it throws, returns a promise that rejects, or calls a callback with an error.
  • Helpers for producing TAP output. Note that I mean strictly for producing the output, and not for running tests or organizing test code.

Here's a sketch of what a super-minimalist test library built on such tools could look like:

import { createTest, TAP } from 'assert'

const tests = {}

export function test(name, fn) {
  tests[name] = createTest(fn)
}

export async function run() {
  const tap = new TAP(process.stdout)
  tap.begin(Object.keys(tests).length)
  process.exitCode = 0
  for (const testName in tests) {
    try {
      await tests[testName].run()
      tap.pass(testName)
    } catch (err) {
      tap.fail(testName, err)
      process.exitCode += 1
    }
  }
  tap.end()
}

[cjihrig]

import { createTest, TAP } from 'assert'

I think it would be best if the testing APIs and assertion APIs were kept separately.

I think it would also be preferable if each test ran in a separate process and globals were not altered (or new globals introduced).

[devsnek]

I think at the core here we should definitely not get complex. rust's test runner really speaks to me. you define some functions that pass on return and fail on panic (throw in js). the test name is just the function name. then we just need a standard tap output and a super minimal human readable output. if you want to get fancy, just compose the test function api with your own api. then you just glob with node --test and node defines the test register function and you're done.

import { test } from 'assert or test or smth';

test(function foo() {
  // ...
});

test(async function bar() {
  // ...
});

this also doesn't force tests to be run in the same process/context/etc. I can see us setting up a new node main context to run each function, especially now that we have snapshots. we don't have to do this though.

[jasnell]

Ok, so from the feedback so far, I think we can answer two specific (and important) questions:

1. Yes, we should add a test runner to core.
2. No, we should not vendor one in but instead create as minimal of a bespoke runner as we can.

Big +1 on keeping it simple.

@cjihrig:

I think it would be best if the testing APIs and assertion APIs were kept separately.

I agree. However, something like import { test } from 'assert/test' would accomplish that. I don't think we should add a new top-level module.

@cjihrig:

I think it would also be preferable if each test ran in a separate process and globals were not altered (or new globals introduced).

I don't think we need every test to be in a separate process. If we stick the the idea that node -test foo.js just runs all the tests that are found in foo.js, then all of those will be run in a single process. If I have a different set of tests that I'd like to run on their own, I just put those in a different file. It's essentially the same as what we do in Node.js own test/parallel/*.

@devsnek :

I think at the core here we should definitely not get complex. rust's test runner really speaks to me. you define some functions that pass on return and fail on panic (throw in js).

Big +1 but I think we do need to have a separate test label. The function name itself is not expressive enough.

import { test } from 'assert/test';

test(() => { /** ... *// }, 'The thing and the other thing do a thing unlike the other other thing');

test(() => { /** ... *//}, 'The thing when modified by this thing, does something else unlike the original thing');

@devsnek:

this also doesn't force tests to be run in the same process/context/etc. I can see us setting up a new node main context to run each function, especially now that we have snapshots. we don't have to do this though.

We also have the option of running each test within a file in its own worker_thread. This can be controlled by the API:

test('/* some test code */', 'a test that runs in a worker', { isolation: 'worker' }); // other values for `isolation` could be 'context', 'process', etc)

@bengl:

TAP output is mostly ubiquitous

Big +1 on just adopting TAP as the output format.

[thebergamo]

Quick note, that for me it feels a bit strange having a "test" lib inside an assertion one, usually we have it in the opposite or at least it's how we're used to.