Skip to content

lean: the type model, subtyping, and a self-checking oracle - #760

Draft
lwshang wants to merge 7 commits into
rewritefrom
lean-core
Draft

lean: the type model, subtyping, and a self-checking oracle#760
lwshang wants to merge 7 commits into
rewritefrom
lean-core

Conversation

@lwshang

@lwshang lwshang commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Adds lean/: a Lean 4 model of Candid's type language and subtyping, as both a
relation and an executable decision procedure, with a self-checking binary that CI
runs. Additive only — nothing on master references any of it.

What is here

Candid/Types.lean Slot, Composite, TypeTable, ClosedType, well-formedness
Candid/Hash.lean the normative field-id hash
Candid/Subtype.lean decSubtype, the decision procedure
Candid/SubtypeSpec.lean Subty/SubtyC, the same rules as a coinductive relation
Main.lean the oracle binary: 85 checks, nonzero exit on any disagreement
.github/workflows/lean.yml builds and runs it (§5's CI carve-out)

lake test runs the oracle.

The three decisions that shaped it

The type table is the only recursion. A table entry is a Composite, its children
are Slots, and a Slot is a primitive or an index — never an inline composite. That
is the wire format's own shape (spec/Candid.md:1208), and the spec draws the
conclusion the model is built on: "Because recursion goes through T, this format by
construction rules out non-well-founded definitions like type t = t."

It pays for itself in the procedure. The only way to recurse is through a pair of
references, so decSubtype carries seen — the pairs this path has assumed — and
descends by recording one. Termination is remaining, the count of pairs seen does
not record; it is named only by termination_by, so the |A| × |B| pair space is
never built at run time. No fuel, no Option Bool, no "unanswered" state in the API,
and the two facts the measure needs are produced by the branch that makes the
decision. The one general lemma required is countP_cons_lt; no mathlib.

Subtyping relates two type tables, not one. A Slot holding a ref means nothing
without its table, and the case that matters most compares a table that arrived on the
wire against the receiver's own type graph. rust/candid/src/types/subtype.rs:19 takes
a single env for both, which works only because callers merge tables first.

The spec's negative premises are eliminable. Two of the four opt rules carry
negative premises, and a non-monotone rule functional has no greatest fixed point — so
the relation would not be definable coinductively at all. Pairing each negative rule
with its positive counterpart collapses them: t <: opt t' holds for every t and
t', which is what subtype.rs:293 implements as a warning-only catch-all.

Two bugs the model found in itself

The memo did not swap with the tables. The function rule's parameter premise is
contravariant, so it swaps the two tables — but it passed seen through unchanged, so
a memo of (A-index, B-index) pairs was read as (B-index, A-index). Since subtyping is
not symmetric, that answered a transposed question from the memo and reported <: for
unrelated types. The contra checks are the witness and fail without the fix.

decSubtype was roughly cubic. An earlier iteration carried the complement of
the memo, seeded with all |A| × |B| pairs and narrowed with List.erase, which
copies up to the erased element. Compiled, on a table of n entries in one cycle:
23 ms at n=100, 370 ms at n=300, 14.3 s at n=1000 — and a 1000-entry type table is
2–3 KB on the wire. Recording assumed pairs instead of carrying their complement makes
the same query <10 ms, 70 ms at n=10,000, 1.6 s at n=50,000, with no stack overflow at
400,000.

What it is checked against

  • 85 checks in the oracle, covering primitives and the numeric tower's lack of
    width subtyping, the four opt rules collapsed, records, variants, functions in
    both variance directions, services, recursive types across two independent tables,
    and well-formedness.
  • A differential against the pre-flat iteration of the same model: 50,000 random
    flat type pairs (expressible in both representations, so the answers must agree
    exactly) and 20,000 random nested pairs flattened into the new form. Zero mismatches
    in answers and in well-formedness.
  • Properties over 20,000 random well-formed types: reflexivity, t <: reserved,
    empty <: t, and transitivity on every triple that satisfied both premises.

Well-formedness

References resolve, labels do not repeat, and two rules the shape of a Composite
does not enforce on its own: a oneway function has no results
(spec/Candid.md:211), and a service's method type denotes a function
(spec/Candid.md:1223) — the only rule that has to look through the table. The 2^32
bound on parameter lists (spec/Candid.md:209) is deliberately not checked, with the
reason recorded: it cannot be violated by anything that fits in memory, but it is what
keeps indexedFrom's UInt32 argument labels distinct.

Deliberately not here

  • A type generator and conformance-vector I/O — Tiers 1–2 of REWRITE.md §3, and
    the next PR from its own branch. The generator used for the differential above lives
    outside the repo today, which is exactly why it should land as a module.
  • Coercion, the binary wire format, the cost model, textual value syntax.
  • Proofs. decSubtype_iff is stated as prose next to the relation, not as a sorry.
  • Verso.

Files outside lean/

REWRITE.md (§5's policy boundary and what slice 1 disproved), crates/README.md and
crates/CLAUDE.md (the candid_types naming table now records Slot and
Composite), and .github/workflows/lean.yml, which §5 explicitly allows to ride the
working branch so that it runs while the code is being written.

lwshang and others added 7 commits August 14, 2026 10:18
… oracle

First code of the rewrite. A Lean 4 model with no mathlib dependency, pinned to
v4.32.2, building a library and an executable that checks itself and exits nonzero
on disagreement -- so CI verifies behaviour rather than only that the model
compiles. 71 checks pass.

Types are finite, with recursion through an explicit TypeTable. MiniCandid
represents types as a CoInductive T; Lean 4 accepts `coinductive` only for
predicates, and the infinite-tree representation cannot be executed anyway. The
table is what the binary format does and what candid_types is specified to do with
arena indices.

Subtyping is written twice, deliberately. Candid/SubtypeSpec.lean holds the
coinductive relation, mirroring spec/Candid.md rule for rule; Candid/Subtype.lean
holds the decision procedure. The theorem connecting them is stated in prose and
is the next slice's first obligation. Keeping them apart is what makes the model a
specification and an oracle at once, rather than one or the other.

Three things the spec turned out to say that the design had to follow:

- The negative premises in the four `opt` rules are eliminable. Pairing each
  negative rule with its positive twin collapses them to `t <: opt t'` for every t
  and t'. This is not a shortcut -- a rule functional with negative premises is
  non-monotone and has no greatest fixed point, so the relation would not be
  coinductively definable at all.
- `principal` is a <primtype> (spec:80), not a reference type. It lives in Prim,
  which also makes `principal <: principal` follow from prim reflexivity.
- The type table may only hold composite types (spec:1227, "no <primtype>"), which
  is stronger than the no-bare-references invariant it replaces and rules out
  unbounded reference walks for a reason that is cited rather than derived.

Subtyping relates two tables, not one. A TypeExpr holding a ref means nothing
without its table, and the case that matters most compares a wire type table
against the receiver's own type graph. Separating them forces the memo onto
reference pairs -- bounded by |A|x|B|, where unfolded expressions are not -- and
exposes that func contravariance swaps the tables along with the types.

Deferred and named in lean/README.md rather than left implicit: `fuel` bounds
recursion depth instead of a termination measure, and returns none rather than
false when exhausted so the model never reports an answer it did not compute;
decSubtype_iff; productive-recursion checking; Verso.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… disproved

Three corrections that writing the first slice forced, none of which were visible
from the plan alone.

**The policy boundary was in the wrong place.** §5 said "only code churn lives on
the branch" and listed directory reservations as policy, which made every README
inside a reserved directory a master-PR item -- so recording a design decision cost
a review round trip. A decision that has to wait for a policy review is a decision
that gets made in someone's head and written down later, or not at all. The
boundary is now location: everything under lean/, crates/ and conformance/ belongs
to the working branch, including their README.md and CLAUDE.md; policy is what sits
outside them. Location is decidable from a diff, where "is this policy?" is
arguable.

Two claims in §5 were also simply false once we started editing: "existing files
are not modified" and a review criterion of "touches nothing existing". Both now
name the real constraint -- nothing under rust/, spec/, test/, coq/ or tools/ is
touched, and REWRITE.md is in scope because it is where the plan gets corrected.

**§6's coq/ deletion condition was wrong on both halves.** It asked that Lean
reproduce every MiniCandid theorem and that the two models be diffed, which treats
them as the same kind of artifact differing in coverage. MiniCandid is a
justification device: it shows that non-obvious design decisions are sound and that
a proposed spec change can be accommodated. The Lean model answers what happens to
a given input. So the condition is that each theorem's purpose is discharged, and
there is no structural diff to perform once one side has finite types and an
explicit table. The scoped nine-constructor comparison survives as a finding worth
hunting, not a gate.

**The layering diagram named a structure that does not exist.** candid_types listed
TypeEnv, but rust/'s TypeEnv is a BTreeMap<String, Type> -- a name-keyed
environment of .did declarations, not an index-keyed table, and both will exist.
The diagram now names TypeExpr/TypeTable/TypeRef/FieldId/ClosedType, matching the
Lean identifiers so that "candid_subtype reads as a transcription of its Lean
counterpart" is checkable rather than aspirational, with a naming table in
crates/README.md and the rule restated in crates/CLAUDE.md. TypeEnv stays reserved
for candid_syntax, where "environment" is the accurate word.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mise

The function rule's parameter premise is contravariant, so it swaps the two type
tables -- but it passed `seen` through unchanged. A memo of (A-index, B-index)
pairs was then read as (B-index, A-index), which asserts a transposed and
generally different question, since subtyping is not symmetric. The result was
`<:` reported for unrelated types.

The new `contra` checks in Main.lean are the witness: `record { f : func (vec nat)
-> () }` against `record { f : func (vec text) -> () }` reaches its parameter
premise with `(1, 2)` in the memo, reads it transposed, and answers `<:` where the
answer is `!<:`. Both checks fail without the one-line fix.

Also records a second gap, which this does not fix. `T.0 = vec (vec T.0)` makes
`ref 0` and `vec (ref 0)` the same infinite type, but every state on that cycle
has a reference on exactly one side, so the reference-pair memo never fires and
no budget suffices. That makes the completeness half of the `decSubtype_iff`
obligation false of the current procedure rather than merely unproved, so the
comments claiming a termination measure exists are corrected too.

The two `vecOmega` checks state the answer the model owes and are marked as known
gaps: reported, not build failures -- and a known gap that starts passing is
itself a failure, so the fix cannot land unnoticed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A table entry is now a `Composite`, its children are `Slot`s, and a `Slot` is a
primitive or an index -- never an inline composite. That is the wire format's own
shape rather than an invention of this model (`spec/Candid.md:1207`), and the spec
draws the conclusion the change is built on: "Because recursion goes through `T`,
this format by construction rules out non-well-founded definitions like
`type t = t`."

What it buys:

- The subtype procedure can only recurse through a pair of *references*; every
  other slot pair is decided outright. So it carries `todo` -- the reference pairs
  it has not yet assumed -- descends by removing one, and `todo.length` is a
  termination measure. The membership test that decides the branch is exactly the
  fact the measure needs, so no invariant is threaded through the recursion.
- `fuel`, `budgetFor`, `TypeExpr.depth` and the whole `Verdict` (`Option Bool`)
  layer are gone. `decSubtype` returns `Bool` and is total, and the public API has
  no "unanswered" state to explain.
- The two `vecOmega` checks recorded as known gaps in the previous commit now pass.
  A cycle whose states hold a reference on alternating sides used to dodge the
  reference-pair memo entirely; flat, the inner composite has to be an entry of its
  own, so the same cycle passes through `(0, 1)` and `(1, 0)` and stops.
- Both `mutual` blocks in the type language collapse: well-formedness is now a
  per-entry check over one flat node, and "the type table may only contain
  composite types" (`spec/Candid.md:1227`) holds by construction. That rule still
  has force at the decoder, which must reject a primitive opcode in an entry
  position.
- `Subty` splits into two mutually coinductive predicates, `Subty` on slots and
  `SubtyC` on composites, mirroring the procedure. `decSubtype_iff` can now be
  stated with no side condition about a budget.

What it costs: a type means nothing without its table, and even `vec nat` needs an
entry, so hand-written types are built with `intern`/`close`. The `.did` surface
syntax is nested, so the parser will need a flattening pass -- which is also what an
encoder does when it emits a type table.

Verified beyond the 79 checks by differencing against the previous model over
250,000 random pairs -- 200,000 flat (expressible in both representations, so the
answers must agree exactly) and 50,000 nested and flattened. Zero mismatches in
answers and in well-formedness.

Also: `ClosedType.ofPrim` replaces `ofSlot`, so the helper cannot build a type whose
root is a dangling reference; `simp_wf` is dropped from the termination proofs,
unnecessary since Lean 4.12; and a dangling reference is no longer handed the top or
bottom rule without being resolved first, which `<:` reported without looking would
be the dangerous direction for a compatibility gate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…n crates/

The module was named after a type it no longer defines. It holds `Slot`,
`Composite`, `TypeTable` and `ClosedType`, so `Types.lean` is what it is.

`crates/README.md` and `REWRITE.md` carried the same stale name in the planned
`candid_types` layer, still describing `TypeExpr` as "one structural node; may
contain references" -- the shape the model dropped. Identifiers are shared between
`lean/` and `crates/` precisely so that "candid_subtype reads as a transcription of
its Lean counterpart" stays checkable, so the naming table now records `Slot` and
`Composite` with the reason neither takes the `Type` prefix: neither is a type, and
both are named after the grammar position they occupy in the spec.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`decSubtype` was roughly cubic. The procedure carried `todo`, the reference pairs
not yet assumed, seeded with all `|A| x |B|` of them and narrowed by
`todo.erase (i, j)` at each descent -- and `List.erase` copies the list up to the
erased element. Compiled, on a table of `n` entries in one long cycle:

    n=100    23 ms
    n=300   370 ms
    n=1000   14.3 s
    n=2000   >100 s

A 1000-entry type table is 2-3 KB on the wire, so that is the same shape as a
cheap-bytes/expensive-work bug, in the artifact meant to become a differential
fuzzing oracle.

The complement was only ever there to be a termination measure. So the procedure now
carries `seen` -- the pairs this path *has* assumed, one cons per descent, membership
by a scan no longer than the path -- and the measure moves into `remaining`, which
counts the pairs `seen` does not record. `termination_by` measures are erased, so the
pair space is never built when the procedure runs. Same total process time on the
same tables:

    n=1000   <10 ms      n=10000   70 ms
    n=50000  1.6 s       n=400000  113 s, and no stack overflow at that depth

The two facts the measure needs are produced by the branch that makes the decision:
the guard says the pair is fresh, and the table lookups say its indices are in range.
So there is still no invariant threaded through the recursion. `remaining` counts
both orientations of the pair space, which is what makes the function rule's table
swap leave the measure alone -- proved as `remaining_swap` rather than assumed.

The one general lemma this needs is `countP_cons_lt`: a `countP` over a list drops
when the predicate flips to `false` on one member that is present, and nowhere gains.
That is the whole of the proof obligation; no pigeonhole argument and no mathlib.

Verified unchanged by differencing against the pre-rewrite model again: 50,000 random
flat pairs and 20,000 random nested pairs, zero mismatches, plus reflexivity, top,
bottom and transitivity over 20,000 random well-formed types.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**Well-formedness.** The model accepted two kinds of type the spec forbids:

    oneway with a result:        wellFormed = true    -- spec/Candid.md:211
    service with a prim method:  wellFormed = true    -- spec/Candid.md:1223

`Composite.annotsOk` covers the first. The second needs the table, since a method's
type is a slot like any other, so it lives in `TypeTable.wellFormed` as
`methodsDenoteFuncs` -- the only well-formedness rule here that looks through the
table. Six checks pin both, including the cases that must stay well formed (`query`
with a result, `oneway` without one, a method that is a function).

One rule is left out on purpose, with the reason recorded: "The list of parameters
must be shorter than 2^32 values" (`spec/Candid.md:209`) cannot be violated by
anything that fits in memory. It is not idle, though -- `indexedFrom` labels
positional arguments with `UInt32`, which wraps, so that bound is what keeps a
function's argument labels distinct.

**Citations.** `spec/Candid.md:1207` is a bare code fence; the `I` block it introduces
starts at 1208. `spec/Candid.md:1221` is the note about multiple representations, not
the method-type rule, which is 1223. All ten citations in `lean/` were checked against
the spec; the rest were exact.

**Vocabulary.** "Slice" was used seven times and defined nowhere, and REWRITE.md does
not use the word at all. It now has a one-line definition where it first appears -- a
label for what landed in a merge window, applied after the fact -- and the two
forward-looking uses in the sources are gone, since the plan is the coverage checklist
and the tiers in REWRITE.md §3.

**`lake test`.** `testDriver = "oracle"`, so the model checking itself is reachable
the ordinary way. CI keeps running the binary explicitly so the check names show up in
the log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant