Written before the operators below were implemented and before anything was measured with them. The rule at the bottom does not change after the table is seen. If the measurement shows the rule was wrong, that is what the report says.
The bond changes a line the patch added into something that behaves differently and runs the
oracle again. It has five operators, in src/gates/oracle-mutants.ts:
invert a comparison, swap the operands of a non-commutative arithmetic operator, replace a
returned expression with a sentinel, swap the two arguments of a call, drop a no-argument call
whose result the chain uses.
Every one of them needs a particular token on the line. Ordinary code does not carry those tokens.
A guard clause, an assignment, a require, a callback: none of them offers these five operators a
target, mutantOfLine returns null, and the bond reports not-bonded, which is the fourth word
and is never read as held.
Reproduced against the built tree before anything here was written:
npm run build && node -e "
import('./dist/gates/oracle-mutants.js').then(m => {
for (const text of [' if (!this.isValid()) {', ' return this',
\"const v8 = require('v8')\", ' this.ctxStorage = null']) {
const got = m.mutantsOfChangedLines({ changed: [{ path:'src/a.js', addedLines:[{line:10,text}] }] });
console.log(got.length ? 'MUTANT ' + got[0].operator : 'none', '|', text.trim());
}
});"
none | if (!this.isValid()) {
none | return this
none | const v8 = require('v8')
none | this.ctxStorage = null
Both adversarial false greens are that shape. Run through the harness's own diff parser and mutant
planner, iamkun/dayjs#3180 and koajs/koa#1946 produce zero mutants between them across every
line they add. not bonded there is not the tool finding a strong oracle. It is the tool having
nothing to ask.
Not from those two patches. Fitting operators to the cases that motivated them is how the current 3-of-3 on gate 3b became partly in-sample, and it buys a number that describes the fitting.
The design reads the Statement and Declaration productions of the language instead, and asks of
each shape what change to one line of it a test could observe. The left column is what the grammar
allows on a line; the right is what covers it.
| Statement shape | a line of it | covered by |
|---|---|---|
| ExpressionStatement, call | emit(name) |
nothing, until delete-statement |
| ExpressionStatement, assignment | this.zone = zone |
nothing, until replace-assigned-value |
| ExpressionStatement, compound assignment | total += one |
swap-arithmetic-operands, only for -, /, % |
| ExpressionStatement, update | at += 1 |
nothing, until delete-statement |
| LexicalDeclaration with initializer | const v8 = require('v8') |
nothing, until replace-assigned-value |
| IfStatement head | if (!this.isValid()) { |
nothing, until negate-condition |
| IfStatement head with a comparison | if (at === last) { |
invert-comparison |
IterationStatement head, while |
while (queue.length) { |
nothing, until negate-condition |
IterationStatement head, for |
for (const one of all) { |
nothing, and nothing here changes that |
| ReturnStatement with a semicolon | return merged; |
return-sentinel |
| ReturnStatement without one | return this |
nothing, until delete-statement |
| ThrowStatement | throw new RangeError(at) |
nothing, until delete-statement |
| BreakStatement, ContinueStatement | continue |
nothing, until delete-statement |
| ImportDeclaration | import fs from 'node:fs' |
nothing, until delete-statement |
| Call in a chain | parents(this).reverse().forEach(cb) |
drop-chained-call |
| Call with two arguments | assign(target, source) |
swap-call-arguments |
| FunctionDeclaration, ClassDeclaration head | function merge(one, other) { |
nothing, and a line-level change is the wrong instrument |
| SwitchCase label | case 'utc': |
nothing, and a line-level change is the wrong instrument |
| Object property | zone: name, |
nothing, and a line-level change is the wrong instrument |
| Brace or bracket punctuation | }, });, ], |
nothing, and there is no behaviour on the line to change |
Three shapes account for most of what is uncovered, and three operators cover them.
An if or while condition wrapped in !( ... ). The keyword is read at a word boundary that is
not a property access, its condition is the balanced parenthesis after it, and the whole condition
is wrapped rather than a token inside it edited, so } else if (a && b) { and
while (queue.length) { are both one rule.
Equivalence residual: a condition whose two branches do the same thing. That is rare enough, and the reason condition negation is the second operator every mutation tool carries.
The right-hand side of a plain assignment or a declaration with an initializer, replaced by a
sentinel no caller asked for. Plain means a single = that is not part of ==, ===, !=,
!==, <=, >=, => or a compound assignment.
The sentinel is chosen the way return-sentinel chooses it, and for the reason that operator was
narrowed: undefined for anything, and a truthy string where the replaced expression is a literal
whose falsiness is known from its spelling, so this.zone = null does not become
this.zone = undefined and get called a change.
Equivalence residual: an assignment nothing reads.
The line replaced by an empty line. The most general operator there is, and the classic: almost any complete statement can be removed, and a test that asserts on what the statement did will notice.
An empty line rather than a removed line, so every line after it keeps its number. The coverage hit map the bond reads is keyed by line, and a mutant that renumbers the file is a mutant nothing can be shown to have run.
Applies where the line reads as a complete statement, and the reading is lexical: brackets balance
across it and never go negative, it does not end in a position that continues ({, an infix
operator, ,, =>), and it does not begin in one (., ?., ), ], }, :, an infix
operator) or with a keyword that needs the block it heads (else, case, default, catch,
finally, do, try).
Equivalence residual: a statement with no observable effect. This is the operator that carries most of it, which is why it is last in the ordering and first to be cut by the mutant bound.
The lexical reading above is not a parser and cannot be. A line can be balanced, start and end on a token that looks final, and still be the middle of an expression the line before it opened. Deleting it then leaves a file that does not compile, the oracle refuses it, and the bond credits the oracle with a refusal it never made. A syntax error is not evidence about anybody's tests.
So a delete-statement mutant is confirmed to parse before it is used, and confirmed
differentially: node --check is run over the file as the patch left it and over the file with the
line blanked.
- The original parses and the mutant parses: the mutant is used.
- The original parses and the mutant does not: the mutant is discarded, and no observation is recorded for it.
- The original does not parse:
node --checkcannot read this dialect, so it cannot tell a syntax error from a mutant, anddelete-statementproposes nothing for that file.
The third case is the residual, named rather than implied away: node --check reads JavaScript,
including ES module syntax in a .js file, and refuses TypeScript and JSX. In a TypeScript or JSX
file the other seven operators still apply and statement deletion does not. Nothing else here
needs the check, because no other operator can produce a file that does not parse: each of them
replaces a token with a token of the same shape, wraps a balanced region, or removes a balanced
one.
How much work the check does, and whether the claim beside it is true.
node scripts/deletion-parse-rate.mjs offers every line of a real file as an added line and checks
every mutant the planner proposes, not only the deletions, because "no other operator can do this"
is the reason only one is checked at runtime and is therefore a claim rather than a design. Over
koa/lib, dayjs/src and commander/lib, 186 files and 2,702 mutants:
delete-statement 929 proposed 5 do not parse 0.54%
replace-assigned-value 920 proposed 0 do not parse 0.00%
invert-comparison 273 proposed 0 do not parse 0.00%
negate-condition 234 proposed 0 do not parse 0.00%
return-sentinel 171 proposed 0 do not parse 0.00%
swap-call-arguments 105 proposed 0 do not parse 0.00%
swap-arithmetic-operands 59 proposed 0 do not parse 0.00%
drop-chained-call 11 proposed 0 do not parse 0.00%
Each of the five deletions is the middle of an expression the line below it continues, which one
line of context cannot see: return proxy && val, return hostname, return arg.required,
getCommandAndParents(this), (args ? ' ' + args : '');. So the lexical rule is right about 99.5%
of the lines it fires on and the check covers the rest. Without it those five reach an oracle, are
refused as compile errors, and count as five bonds that held.
The claim was false the first time it was measured, and the measurement is why it is true now.
swap-call-arguments broke 1 of its 105 on val.split(...), splitting on a pattern whose own comma
is part of it: withoutLiterals masked strings and comments and had never masked a regular
expression, so the argument scan read that comma as the separator between two arguments and swapped
halves of a regex. The same blindness reached every operator that looks for a token, since a
comparison inside a pattern is not a comparison. A pattern is now masked like any other literal,
and which of two things a slash is gets settled by what precedes it, wrong only in the direction
that masks too much and costs a mutant.
vacuous means the oracle ran a line the patch added and then accepted a change to that same
line, so running it established nothing about that line. Two facts have to be shown for that, and
until now only one of them was.
Shown already: the oracle executed the mutated line, read from the coverage of the oracle's own
run, which is the same reading reach takes. Absent that, the answer is unshown.
Not shown, and the reason this document exists: that the mutant changes behaviour at all. A mutant that changes nothing is indistinguishable from an oracle that failed to notice one that did, and the audit that separated them was a person reading each mutated line. That audit found two equivalent mutants and narrowed two operators. Broadening the operators makes the problem worse, because the general operators are the ones that produce equivalent mutants, and a hand audit does not scale and is not evidence.
So a second detector adjudicates every mutant the oracle accepted on a line it ran, and vacuous
requires that detector to have seen a difference. Two detectors, cheapest first, and the first one
that witnesses a difference decides it:
-
Coverage. The oracle is run again over the mutant with the same instrumentation reach used, and the hit map is compared with the unmutated one. A line other than the mutated line that ran in one and not in the other is a demonstrated difference in what the program did. The mutated line is excluded because a deleted statement stops running by construction and that is not a finding. Presence and absence only, never a change in hit count: a count that moves is the weaker signal and a flaky one, and the direction this can be wrong in is a refusal, which is the direction that must not be wrong.
-
The repository's own suite. The gates that passed with the patch applied are run again over the mutant. One that now fails is a demonstrated difference. A gate that failed with the patch applied witnesses nothing and is not read.
If neither witnesses a difference the verdict is unshown and never vacuous, under the reading
that requires a witness. That is an abstention, it does not refuse certification, and it is not
flattened into held.
Two absences, and they are not one word. none says everything that could be asked was asked and
nothing showed a difference, which includes a patch whose own checks all failed and so leave the
second detector nothing to compare a mutant against. not adjudicated says no detector was asked
at all: the oracle refused the mutant, or nothing showed it ran the line, or the verdict was
already settled. Recording the first as the second put a refusal on two commander rows with
nothing beside it to explain itself.
What each detector can and cannot see, stated because the pair is not a proof of equivalence:
- Coverage sees a mutant that changes control flow or stops reaching code. It does not see one that changes a value along a path that runs either way, which includes changing the order of a sequence every element of which is visited.
- The repository's own suite sees a mutant that breaks behaviour the project already had. It cannot see one that only affects behaviour the patch is adding, because the suite was written before that behaviour existed. On a patch that adds a feature, this detector is silent by construction.
- So a mutant that changes only a value, on a path that runs either way, in code the project did
not have before, is witnessed by neither and lands in
unshown. That is the accepted residual.
The cost is bounded by where it is spent. Only a mutant the oracle accepted on a line it ran can be vacuous, so only those are adjudicated, and the first witnessed difference ends the adjudication for that patch because one vacuous mutant decides the bond. Detector 1 is another oracle run. Detector 2 is another suite run and is capped per patch, with anything past the cap recorded as not adjudicated rather than as either answer.
The operator list is ordered by how often an operator produces a mutant that behaves exactly as the original did, because the bound on mutants per patch cuts the tail of that list first and each mutant is another oracle run. One mutant per line, by the first operator in this order that fires, so two rules never compete for one line.
invert-comparison a flipped comparison changes the predicate whatever the operands are
negate-condition a negated condition changes which branch runs
swap-arithmetic-operands equivalent where the operands are equal, or the operation commutes
return-sentinel equivalent where the caller reads only truthiness
replace-assigned-value equivalent where nothing reads what was assigned
swap-call-arguments equivalent where the callee does not care about order
drop-chained-call equivalent where the dropped call was the identity on its input
delete-statement equivalent where the statement had no observable effect
The bound stays at six mutants per patch and two per operator, unchanged, so the oracle runs a certified patch pays for do not move with this change. Six slots against eight operators is a real constraint and it is a knob: raising it buys a stronger bond and costs oracle runs.
The two adversarial false greens were read before this was written, because reproducing the zero-mutant result required reading them. Neither is out-of-sample evidence for these operators and neither will be reported as though it were.
- In sample.
iamkun/dayjs#3180andkoajs/koa#1946, the two adversarial false greens, andtj/commander.js#1671, which the existing five operators were chosen knowing about. - Out of sample. Every other patch in both corpora, none of which motivated anything here, and the adversarial tasks run after these operators are committed. Adversarial rows produced after the freeze are the only out-of-sample adversarial evidence there is, and they are reported separately from the three names above.
No operator or threshold above is changed in response to what any of these produce. A finding that argues for a change is reported as a finding, and the change is a later commit with its own measurement.
The broadened operators and the adjudication ship together, blocking, if both hold:
- Zero false refusals across both corpora. A false refusal is a
vacuousverdict on a patch both oracles accept whose mutant does not change behaviour, or whose refusal comes from an artifact of the check. This is the condition that vetoes, because it is the one where the tool gets worse. - The known-answer smoke is unchanged.
koajs/koa#1946true-red,koajs/koa#1999andkoajs/koa#1904certified.
If either fails, the broadened operators go in report-only, and the report names the condition and the lines responsible.
tj/commander.js#1671 staying refused is reported, and is deliberately not a condition. The
adjudication may demote it from vacuous to unshown, because its mutant drops a .reverse()
and neither detector can witness that: coverage sees the same lines run the same number of times
in a different order, and commander's own suite never had the precedence this patch adds. If that
happens, the tool has stopped refusing the one false green bonding was built to catch, and the
reason is worth as much as the catch was: that refusal rested on a person reading the mutated line
and calling it behaviour-changing. Invariant 16 says a verdict nothing can re-derive is named as
not re-derived rather than agreed with. Keeping the refusal by making the hand judgement a
condition would be keeping a number by keeping the audit that produced it, which is the practice
this replaces. It is reported either way, with the detector output that decided it.
The rule above ran unchanged over both corpora. This section is what the table showed, written after seeing it and marked as such.
The operators do what they were built to do. All eight fire on the mined corpus, and the three
new ones account for 25 of the 34 mutants built: replace-assigned-value 11, negate-condition 8,
delete-statement 6. Every certified patch now carries a held bond. koa#1999, the patch the
plan called untouchable by any operator here and the one the rejected stricter option would have
cost, certifies on held. The known-answer smoke is unchanged: koa#1946 true-red, koa#1999 and
koa#1904 certified.
The adjudication fired, and it fired correctly. One vacuous verdict across the corpus, on
iamkun/dayjs#3180 in the ordinary arm, witnessed by coverage: negating a guard that returns early
on an invalid date stops the lines after it from running, the sealed oracle accepted that, and the
patch was already refused on reach. Behaviour-changing, demonstrated by an instrument, and not an
artifact.
And it cost the refusal of tj/commander.js#1671, which the pre-registration said it might.
Its drop-chained-call mutant reads unshown with witness: none: both detectors were asked and
neither answered. So the bond is held on the other three mutants, the patch certifies, and it is
a false green again. The mined arm goes from 0 false greens in 4 certified to 1 in 5, and 3b from
3 of 3 to 2 of 3.
Here is the finding. Across the whole corpus the witness requirement changed exactly one verdict, and that one was a true catch. It prevented zero false refusals, because the aggressive operators produced no accepted-and-seen mutant that was equivalent: only two mutants in the corpus were ever adjudicated at all, one witnessed by coverage and one by nothing. The insurance the requirement exists to provide has not paid out once, and its premium is the one false green this corpus still holds.
That is derivable from the record rather than from this paragraph, because a mutant's witness is
recorded beside its verdict: a mutant carrying any witness other than not-adjudicated was
accepted on a line the oracle ran, which is the whole of what the earlier rule needed.
node scripts/bond-cost.mjs prints both regimes off the same rows.
Not a blind pre-registration, and not presented as one: it is proposed because the table above argues for it. What is fixed here before the numbers are read is the rule and the conditions, and nothing below moves once the third column is seen.
Refuse on a mutant the oracle ran and accepted, and record the witness rather than requiring
it. Two facts, kept as two: that the oracle accepted a change to a line it ran, which is what
vacuous has always meant and is true whatever a detector saw, and whether anything showed the
mutant changed the program, which is recorded beside it. A refusal carrying witness: none says
what backs it, so the audit that used to read every vacuous verdict now reads only those, and the
count is published rather than lived with.
Its cost is the one the witness requirement was built to avoid: a refusal on a mutant that changes nothing is a false red. What makes that answerable rather than invisible is the field, which names exactly which refusals could be one.
The switch is one exported boolean beside the one that already decides whether a vacuous bond refuses at all, so either regime is one line and the evidence beside it.
It becomes the default if all three hold:
- Zero false refusals across both corpora, where a false refusal is a patch both oracles
accept, certified under report-only, that this regime refuses on a mutant which does not change
behaviour. Every refusal carrying
witness: noneis read by hand once, which is now a named and countable set rather than every vacuous verdict. - The known-answer smoke unchanged:
koa#1946true-red,koa#1999andkoa#1904certified. tj/commander.js#1671refused, which is the catch this exists to recover.
If any of them fails, the witness requirement stays the default and the report names the condition. Whichever way it goes, both regimes are published side by side, because the choice between them is the finding rather than the number either one produces.
report-only certified 16 of 97 (16.5% [10.4, 25.1]), false green(s) 1: 6.3% [1.1, 28.3]
witness required certified 16 of 97 (16.5% [10.4, 25.1]), false green(s) 1: 6.3% [1.1, 28.3]
witness recorded certified 15 of 97 (15.5% [9.6, 24.0]), false green(s) 0: 0.0% [0.0, 20.4]
node scripts/bond-cost.mjs, all three off the same rows.
The first two rows are the same row. Requiring a witness does not make the bond stricter by
some measured amount; it makes the bond refuse nothing, and the certify rate and the false-green
rate come back identical to the regime where bonding blocks nothing at all. The third row is the
column the pre-registered decision at 343ae4740 published, to the digit. So the choice is not
between two strengths of a check. It is between a check and no check, with the no-check option
reverting an earlier decision without saying so.
All three conditions hold.
- Zero false refusals. The regime refuses one row of the ninety-seven,
tj/commander.js#1671, and its held-back oracle rejects that patch. A refusal of a patch a second oracle rejects is not a false refusal by any reading. No other row moves, so there is no other refusal to audit. - The known-answer smoke is unchanged, and the record shows it directly rather than by
re-running: none of
koa#1946,koa#1999orkoa#1904has a mutant carrying any witness other thannot-adjudicated, so no verdict of theirs can turn on this. They stand at true-red, certified and certified. tj/commander.js#1671is refused.
So the witness becomes a recorded field and the refusal returns to what vacuous has always
meant. vacuousRequiresAWitness is false, one line, both values tested, and reverting the commit
that set it restores the stricter reading.
What that leaves open, which is the thing the stricter reading existed to close. A refusal on a
mutant that changes nothing is a false red, and nothing here proves a mutant changed anything. What
is different from the hand audit it replaces is that the record names which refusals could be one:
a refusal carrying witness: none is the only shape that needs reading, bond-cost.mjs counts
them, and swarm ci prints the witness beside the mutant so a reader weighs the refusal rather
than taking it. On this corpus that set has one member and it is a true catch. It is not zero by
construction and will not stay at one.