ModelingToolkitBase QA lane red since #4832: JET typo mode reports 263 errors, 234 of them from Moshi `@match` codegen

[ChrisRackauckas-Claude]

Summary

The lib/ModelingToolkitBase [QA] / Julia 1 lane of Sublibrary CI has been red on every master run since #4832 (92c27bb9fd, 2026-08-11) introduced SciMLTesting.run_qa(ModelingToolkitBase; jet = true), i.e. JET.test_package(ModelingToolkitBase; target_modules = (ModelingToolkitBase,), mode = :typo). The check reports 263 possible errors and fails; before that commit the QA lane ran only test/qa/jet.jl (54 @test_calls) and the report_package check did not exist, so there is no MTK commit to bisect to.

Latest master run: https://github.com/SciML/ModelingToolkit.jl/actions/runs/33444273161/job/99660311927 (e91f13e4, JET 0.12.1, Julia 1.12.7: JET-test failed ... 263 possible errors found). Every earlier run that executed the job is red as well, e.g. https://github.com/SciML/ModelingToolkit.jl/actions/runs/31478008118 (the #4832 PR itself, merged red), https://github.com/SciML/ModelingToolkit.jl/actions/runs/32557851438/job/96996767096, https://github.com/SciML/ModelingToolkit.jl/actions/runs/33305672292/job/99250653964.

The root tests / QA (julia 1) lane is green because test/qa/Project.toml pins JET = "0.9,0.10,0.11" and analyses only the ModelingToolkit facade; lib/ModelingToolkitBase/test/qa/Project.toml has no JET compat, but the JET version is not the cause (see below).

Local reproduction (clean master worktree, 22cb5b0a21)

export JULIA_DEPOT_PATH="$HOME/sandbox/initwrap/depot:$HOME/.julia"
cd lib/ModelingToolkitBase/test/qa
julia +1.12 --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate(); include("aqua.jl")'

Quality Assurance: JET-test failed at ~/.julia/packages/JET/vOTu0/src/JETBase.jl:1312
  Expression: (JET.report_package)(ModelingToolkitBase; toplevel_logger = nothing, target_modules = (ModelingToolkitBase,), mode = typo)
  ═════ 263 possible errors found ═════
  ┌ default_toterm(x::SymbolicUtils.BasicSymbolicImpl.var"typeof(BasicSymbolicImpl)"{SymbolicUtils.SymReal}) @ ModelingToolkitBase .../src/variables.jl:268
  │ local variable `##And#280#f#1` may be undefined: ##And#280#f#1::Any
  ...
Test Summary:                                | Pass  Fail  Total      Time
Quality Assurance                            |   20     1     21  29m27.9s

Running only the JET call (JET.report_package(ModelingToolkitBase; toplevel_logger = nothing, target_modules = (ModelingToolkitBase,), mode = :typo)) on Julia 1.12.7:

JET	Moshi	reports	JET time
0.12.1	0.3.12	263	1036 s
0.11.6	0.3.12	263 (identical set)	1041 s

So this is not a JET 0.12 regression. Of the 263 reports, 234 are local variable ``##And#N#x#1`` / ``##Call#N#x#1`` may be undefined at Moshi.Match.@match sites (utils.jl, systems/abstractsystem.jl, systems/problem_utils.jl, systems/callbacks.jl, variables.jl, systems/system.jl, ...); the remaining 29 are in hand-written code (see the PR below).

Root cause of the 234 @match reports: Moshi's @match codegen

For Pattern(; f, args) && if f isa Int end => body, Moshi 0.3.12 expands the condition to

if begin
        v = x
        begin
            v isa A && begin
                begin ff = v.f; true end && begin gg = v.g; true end
            end
        end
    end && ff isa Int

A begin ... end whose value is a && makes Julia lower ff through a phi node with an #undef edge, so every later use of ff (the guard, the body's let) gets a throw_undef_if_not and JET's typo mode reports it. The runtime behaviour is correct. This is the pattern already reduced in aviatesk/JET.jl#858 and JuliaLang/julia#62745, and discussed for the root QA lane in #4958. Shapes tested with JET.report_call(f, (Any,); mode = :typo) on Julia 1.12.7 / JET 0.12.1:

shape	reports	throw_undef_if_not in code_typed
Moshi 0.3.12 output (blocks around each conjunction)	3	yes
same, but no block around the conjunctions	3	yes
holder assignment hoisted, blocks kept	3	yes
flat && chain, (begin v = x; true end) && v isa A && (begin ff = v.f; true end) && ... && ff isa Int	0	no
flat chain, left-nested ((a && b) && c)	0	no

Fix candidate (upstream, Moshi.jl)

Emitting the match condition as one flat && chain fixes it: https://github.com/ChrisRackauckas-Claude/Moshi.jl/tree/flat-match-conditions (commit ef79026, 3 small changes in src/match/emit/{ctx,logic,call}.jl plus a regression test asserting no throw_undef_if_not in the typed IR of guarded matches). With it:

• the regression test fails on v0.3.12 and passes on the branch, on Julia 1.10.11 and 1.12.7; Moshi's full test suite passes on both;
• JET.report_call on real @match functions goes from 3/2/5/7 reports to 0/0/0/0;
• JET.report_package(ModelingToolkitBase; ...) on master goes from 263 to 29 reports and from 1036 s to 332 s.

Roger-luo/Moshi.jl is not a SciML repository, so no PR was opened there; the branch is ready to open if you want it.

The other 29 reports are real

They are fixed in the PR linked in the first comment below: Shift(steps) calling new from an outer constructor, s/sz/expr/v typos, an undefined ModelingtoolkitizeParametersNotSupportedError, extend calling the removed convert_system, the Substituter method for AffectSystem using removed System fields, optext/coeff undefined on some paths, and a handful of locals that are only bound on some branches or captured by closures after reassignment. With that PR and the Moshi branch together, report_package reports 0 (JET run time 330 s instead of 1036 s).

Also noticed

• With Julia 1.10.11 the same env resolves JET 0.9.18 and report_package produces 38 unrelated toplevel reports (UndefVarError: MassActionJump not defined, ConstantRateJump, BlockedArray, BandedMatrix, ...), i.e. JET 0.9's virtual processing of using X: name imports. test_groups.toml lists versions = ["lts", "1"] for QA but Sublibrary CI only ran a Julia 1 QA job, so this is not currently exercised.
• Moshi's plain-struct keyword pattern A(; f) expands to Core.getfield(v, f) with f unquoted (QuoteNode missing in src/match/emit/call.jl), so it throws UndefVarError: f. MTK only uses keyword patterns on @data variants, which are handled correctly.

[ChrisRackauckas-Claude]

The 29 hand-written findings are fixed in #5065 (draft). Numbers from that PR, Julia 1.12.7 / JET 0.12.1: master 263 reports; PR alone 234 (all ##And/##Call Moshi temporaries); PR + Moshi branch https://github.com/ChrisRackauckas-Claude/Moshi.jl/tree/flat-match-conditions 0 reports, JET time 330 s instead of 1036 s.

[ChrisRackauckas-Claude]

Reopening: the QA lane is still red on master. #5065 fixed the 29 hand-written findings, but the 234 Moshi @match reports remain, and the fix for those (Roger-luo/Moshi.jl#93) has not merged — the newest Moshi release is still 0.3.12, which does not contain it.

Independently reproduced today (2026-09-07)

Fresh clone, 542ad62fe5, Julia 1.12.7, Moshi 0.3.12 / JET 0.12.1 / SciMLTesting 2.13.0:

$ GROUP=QA julia +1 --project=. -e 'using Pkg; Pkg.test()'
Quality Assurance: JET-test failed at JET/src/JETBase.jl:1312
  ═════ 234 possible errors found ═════
Test Summary:      | Pass  Fail  Total      Time
Aqua Tests         |   20     1     21  18m24.5s
  Quality Assurance|   20     1     21  18m11.4s
ERROR: Package ModelingToolkitBase errored during testing

All 234 are local variable ... may be undefined, and every one carries a Moshi gensym prefix — 182 ##And#, 52 ##Call#. No other report kind appears.

Never green

The JET check was enabled by #4832 (jet = true, 92c27bb, 2026-08-11) and failed on that very run. CI logs for the ModelingToolkitBase [QA] job:

• 31471637625 (8a2be22, the commit immediately before): QA red, but no JET-test failed in the log — it was failing on Aqua/ExplicitImports only.
• 31478305489 (92c27bb, Adopt SciMLTesting 2.4 strict QA #4832): JET-test failed, ═════ 263 possible errors found ═════, 249 of them may be undefined.

Every ModelingToolkitBase [QA] job on master from 2026-08-03 through today is failure (checked 120 Sublibrary CI runs). The JET portion has never passed since it was turned on.

Not a dependency regression

The count is unchanged across the whole compatible dependency range. 15-line MWE with no SciML packages:

using Moshi.Data: @data
using Moshi.Match: @match
using JET

@data Foo begin
    A(Int)
    B(Int, Int)
end

f(x::Foo.Type) = @match x begin
    Foo.A(a) && if a > 0 end => a
    Foo.B(a, b) => a + b
    _ => 0
end

@report_call f(Foo.A(1))   # 4 reports

┌ f(x::Main.Foo.var"typeof(Foo)") @ Main
│ local variable `##And#283#a#1` may be undefined: ##And#283#a#1::Int64
│ local variable `##Call#286#a#1` may be undefined: ##Call#286#a#1::Int64
│ local variable `##Call#286#b#1` may be undefined: ##Call#286#b#1::Int64

4 reports on Moshi 0.3.7, 0.3.8, 0.3.9, 0.3.11, 0.3.12 and on JET 0.11.6, 0.12.0, 0.12.1. git grep -c @match lib/ModelingToolkitBase/src is 61 at both 92c27bb and 542ad62 — MTKBase's own @match usage did not change either.

One report is emitted per field binding in a @match constructor pattern, guard or no guard:

pattern	reports
Foo.A(a) => a; _ => 0	1
Foo.A(a) => a (single clause)	1
Foo.B(a,b) => a+b; _ => 0	2
Foo.A(a) && if a>0 end => a; _ => 0	1
Foo.A(_) => 1; _ => 0	0

False positives, but not a JET artifact

The code cannot throw: Moshi emits isa_variant(v, A) && begin a = variant_getfield(...); true end && a > 0, so a is read only when the assignment ran. f(Foo.A(5)), f(Foo.A(-5)) and f(Foo.B(2,3)) all return correctly.

However JET is reporting faithfully — Julia's own optimizer leaves the check in:

ci = only(code_typed(f, (Foo.Type,); optimize=true))[1]
occursin("throw_undef_if_not", sprint(show, ci))   # true

Hand-written analogues of the same shape (use in condition, use in a let in the branch body, two sequential bindings, binding from an opaque call) all optimize to undef_check=false. It is specifically Moshi's nesting — wrapping each conjunction in a begin ... end whose value is a && — that leaves a maybe-undefined phi the optimizer will not fold. That is what Roger-luo/Moshi.jl#93 flattens, and the upstream tracking issues are JuliaLang/julia#62745 and aviatesk/JET.jl#858.

Verified the fix independently: with Moshi#flat-match-conditions, every row of the table above goes to 0 reports.

Blocked

Nothing on the MTKBase side can clear this — no open MTK PR touches test/qa/aqua.jl. The lane stays red until Moshi#93 merges, a Moshi release ships it, and lib/ModelingToolkitBase/Project.toml floors Moshi at that release (currently Moshi = "0.3.7", so an unfloored fix release would not be guaranteed in CI's resolve).

Until then the QA job is a permanently red check that reviewers have to ignore, which also masks any genuine JET regression. Worth deciding explicitly whether to leave it red or gate it behind jet_broken = true (SciMLTesting auto-flags an Unexpected Pass once it goes clean, so it would self-announce when Moshi lands).

🤖 Generated with Claude Code

https://claude.ai/code/session_01H5RuLDtEazwTKPkNbPeZVu