allow `bail` setting to control when to bail out of a failing test run by bookman25 · Pull Request #7335 · jestjs/jest

bookman25 · 2018-11-06T20:52:58Z

Summary

Sometimes when running a full suite of tests there might be a config change (or some other failure) that leads to a catastrophic failure. Instead of continuing to run the entire suite, have an early exit after say 15 failures, something must be seriously wrong so abort the run.

Typically in CI mode we'll want to try to let the tests run so we can see all the failures at one time (which is why bail is not quite enough). But in the event of everything failing there's no need to continue running. We had a case where a branch of code was generating 100GB worth of logs because of test failures, with this change it would have been able to early abort.

Test plan

Unit tests

SimenB · 2018-11-06T22:37:39Z

Interesting idea! I wonder if we could overload bail (making true mean 1) instead of adding a new option?

@thymikee @rickhanlonii

thymikee · 2018-11-07T07:59:40Z

I like passing a number to bail so we don't have to introduce yet another config option.

SimenB · 2018-11-08T22:18:51Z

 For stability and safety reasons, only part of the global configuration keys can be updated with `updateConfigAndRun`. The current white list is as follows:

- [`bail`](configuration.html#bail-boolean)
+- [`bail`](configuration.html#bail)


does the link work? You can run yarn start in the website folder to test links etc

Maybe I'm just doing something wrong, but I've tried yarn start and yarn build then yarn start in /website/ and it doesn't seem to be reflecting any of the changes from these *.md files.

Ok, I was able to find the link in /build folder. Updated to match now.

You need to open the next docs, by default it renders the versioned docs. No need to build :)
You can select the version by clicking the version number in the upper left

This document gives a 404, though, can yiu double check locally?

https://deploy-preview-7335--jest-preview.netlify.com/docs/en/next/watch-plugins

So the link itself for configuration#bail-number-boolean seems to work correctly. But the link you posted, as well as when running locally, and going to /next/watch-plugins both seem to return page not found. Seems to be partially broken in master: https://jestjs.io/docs/en/next/watch-plugins Looks like it's because of original_id inside the header. I've removed it because I don't see any references to that property outside of versioned docs and removing it seems to fix that link so we can test the updated link.

bookman25 · 2018-11-26T15:49:08Z

@SimenB anything else? I believe I've addressed all of the outstanding review comments.

thymikee · 2018-11-26T18:25:39Z

-    if (this._globalConfig.bail && aggregatedResults.numFailedTests !== 0) {
+    if (
+      this._globalConfig.bail !== 0 &&
+      aggregatedResults.numFailedTests >= this._globalConfig.bail


this will still be true if bail is explicitly set to false. Shouldn't we normalize it just in case?

There shouldn't be any case where bail gets to this point that isn't a number: https://github.com/facebook/jest/pull/7335/files#diff-7121e12d406ff9a3060a13ee808632de

thymikee · 2018-11-28T14:31:13Z

+          value = options[key] ? 1 : 0;
+        } else if (isNaN(options[key])) {
+          value = 1;
+          argv._.push(options[key]);


can't we just remove this line? looks fishy

bookman25 · 2018-12-05T12:08:42Z

@thymikee @SimenB any outstanding concerns with this approach?

thymikee

👍

thymikee · 2018-12-05T13:04:52Z

@rickhanlonii you have anything to add here? if not, feel free to merge :)

codecov-io · 2018-12-06T12:56:17Z

Codecov Report

Merging #7335 into master will increase coverage by 0.29%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master    #7335      +/-   ##
==========================================
+ Coverage   67.15%   67.44%   +0.29%     
==========================================
  Files         247      247              
  Lines        9502     9511       +9     
  Branches        5        5              
==========================================
+ Hits         6381     6415      +34     
+ Misses       3119     3094      -25     
  Partials        2        2

Impacted Files	Coverage Δ
packages/jest-config/src/ValidConfig.js	`100% <ø> (ø)`	⬆️
packages/jest-config/src/Defaults.js	`100% <ø> (ø)`	⬆️
packages/jest-config/src/normalize.js	`83.13% <0%> (-2.41%)`	⬇️
packages/jest-cli/src/TestScheduler.js	`73.38% <100%> (+8.06%)`	⬆️
packages/jest-cli/src/lib/update_global_config.js	`93.75% <75%> (-1.91%)`	⬇️
packages/jest-cli/src/ReporterDispatcher.js	`87.5% <0%> (+12.5%)`	⬆️
packages/jest-cli/src/testResultHelpers.js	`72.72% <0%> (+63.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a2237d...c807bb4. Read the comment docs.

rickhanlonii

LGTM just a couple questions!

rickhanlonii · 2018-12-06T16:47:26Z

Thanks @bookman25, I'll merge when we're green

rickhanlonii · 2018-12-06T17:48:18Z

Thanks @bookman25!!

* master: (24 commits) Add `jest.isolateModules` for scoped module initialization (jestjs#6701) Migrate to Babel 7 (jestjs#7016) docs: changed "Great Scott!" link (jestjs#7524) Use reduce instead of filter+map in dependency_resolver (jestjs#7522) Update Configuration.md (jestjs#7455) Support dashed args (jestjs#7497) Allow % based configuration of max workers (jestjs#7494) chore: Standardize filenames: jest-runner pkg (jestjs#7464) allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) Add issue template labels (jestjs#7470) chore: standardize filenames in e2e/babel-plugin-jest-hoist (jestjs#7467) Add node worker-thread support to jest-worker (jestjs#7408) Add `testPathIgnorePatterns` to CLI documentation (jestjs#7440) pretty-format: Omit non-enumerable symbol properties (jestjs#7448) Add Jest Architecture overview to docs. (jestjs#7449) chore: run appveyor tests on node 10 chore: fix failures e2e test for node 8 (jestjs#7446) chore: update docusaurus to v1.6.0 (jestjs#7445) Enhancement/expect-to-be-close-to-with-infinity (jestjs#7444) Update CHANGELOG formatting (jestjs#7429) ...

jestjs#7335)

* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <amodestduck@gmail.com> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files

jestjs#7335)

* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (jestjs#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <amodestduck@gmail.com> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files

github-actions · 2021-05-12T05:14:15Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

facebook-github-bot added the cla signed label Nov 6, 2018

SimenB reviewed Nov 7, 2018

View reviewed changes

Comment thread CHANGELOG.md Outdated

bookman25 changed the title ~~add bailAfter setting to control when to bail out of a failing test run~~ allow bail setting to control when to bail out of a failing test run Nov 8, 2018

SimenB requested changes Nov 8, 2018

View reviewed changes

SimenB requested review from rickhanlonii and thymikee November 26, 2018 16:20

SimenB approved these changes Nov 26, 2018

View reviewed changes

thymikee reviewed Nov 26, 2018

View reviewed changes

thymikee reviewed Nov 28, 2018

View reviewed changes

thymikee approved these changes Dec 5, 2018

View reviewed changes

Comment thread packages/jest-config/src/normalize.js

rickhanlonii approved these changes Dec 6, 2018

View reviewed changes

Comment thread docs/CLI.md Outdated

Comment thread packages/jest-cli/src/cli/args.js

allow bail setting to control when to bail out of a failing test run

60cb6e0

rickhanlonii merged commit 3e805bc into jestjs:master Dec 6, 2018

willsmythe pushed a commit to willsmythe/jest that referenced this pull request Dec 20, 2018

allow bail setting to control when to bail out of a failing test run (

1ebc5c4

jestjs#7335)

captain-yossarian pushed a commit to captain-yossarian/jest that referenced this pull request Jul 18, 2019

allow bail setting to control when to bail out of a failing test run (

8e30fd0

jestjs#7335)

github-actions Bot locked as resolved and limited conversation to collaborators May 12, 2021

Uh oh!

Uh oh!

Conversation

bookman25 commented Nov 6, 2018

Summary

Test plan

Uh oh!

SimenB commented Nov 6, 2018

Uh oh!

Uh oh!

thymikee commented Nov 7, 2018

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bookman25 commented Nov 26, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bookman25 commented Dec 5, 2018

Uh oh!

thymikee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thymikee commented Dec 5, 2018

Uh oh!

codecov-io commented Dec 6, 2018

Codecov Report

Uh oh!

rickhanlonii left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rickhanlonii commented Dec 6, 2018

Uh oh!

rickhanlonii commented Dec 6, 2018

Uh oh!

github-actions Bot commented May 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants