Can we test code coverage of AD testing?

[seabbs]

Not clear if there is a way to do this? Maybe check DifferentiationInterfactTest to see if this has been solved. The reason is that without it is hard to know if a package does actually test all its functionality for each AD backend.

[seabbs]

Compatibility with direct automatic differentiation algorithms (ForwardDiffSensitivity, ReverseDiffAdjoint, etc.) can be queried using the SciMLBase.isautodifferentiable(::SciMLAlgorithm) trait function.

This is the system that SciMl uses to track DI support it appears

[seabbs-bot]

Following the upstream answer from @gdalle on JuliaDiff/DifferentiationInterface.jl#1024:

All DIT scenarios do is run your own code through AD systems, so any lines of your own function that get executed will be covered. Perhaps the issue is that your docs and benchmark CI runs don't get added to the statistics computed by Codecov on your final README badge?

That is exactly our situation, so there is nothing AD-specific to build. The AD suite already exercises real source lines; we just never collect or upload coverage from it.

Why there is a gap today

codecoverage.yaml only runs the main suite (julia-actions/julia-runtest, default project, test/runtests.jl).
The AD gradient suite lives in a separate project (test/ad, run by the ad-*.yaml per-backend workflows) and none of those collect or upload coverage.
So the AD-specific code, the five ext/CensoredDistributions*Ext.jl rule files plus helpers like _gamma_cdf and _grad_p_a_series, only counts as covered insofar as the main tests happen to load it; the dedicated gradient tests contribute nothing to the number.

Decision: a coverage flag and badge per backend

Coverage is not the same across backends: each one runs a different ext/ rule file (Enzyme scenarios execute CensoredDistributionsEnzymeExt.jl, Mooncake the Mooncake ext, and so on) and can take different src/ branches. A single combined upload would report only the union and hide those differences, so we will instead emit one Codecov flag per backend, each with its own coverage badge, mirroring the existing per-backend test badges. This is closest to the original question here: whether each backend exercises all the functionality.

Plan, per ad-*.yaml backend job:

1. Run instrumented: julia --project=test/ad --code-coverage=user test/ad/runtests.jl <tag>.
2. Process src/ and ext/: using Coverage; LCOV.writefile("lcov-ad.info", vcat(process_folder("src"), process_folder("ext"))).
3. Upload with a backend flag: codecov/codecov-action@v6 with e.g. flags: ad-enzyme-reverse; tag the existing main upload flags: unit.
4. Add a codecov.yml declaring all flags with carryforward: true, so each workflow only updates its own flag and the project total is the merge.

Per-backend badges then come from badge.svg?flag=ad-enzyme-reverse, etc.

Local parity

Add a task coverage-ad mirroring the existing task coverage, so the same lcov-ad.info can be inspected locally without waiting on CI:

julia --project=test/ad --code-coverage=user test/ad/runtests.jl
julia --project=test -e 'using Coverage; cov = vcat(process_folder("src"), process_folder("ext")); LCOV.writefile("lcov-ad.info", cov)'

(Run without a tag for the full union, or pass a backend tag to scope it.)

Does this give correctness, not just execution?

Largely yes. Every scenario runs through DIT.test_differentiation(...; correctness = true) against a ForwardDiff reference (test/ad/setup.jl), and the broken-lists are empty, so every covered line executed inside a differentiation whose end-to-end gradient was verified. The per-backend flags therefore read as "fraction of code on a correctness-checked AD path for this backend", not merely "executed".

Two limits keep it honest:

• Line-execution is not a local-partial proof. Coverage records that the primal line ran; DIT checks the aggregate gradient. A covered line's derivative can still be wrong if it is masked, cancels, or is zero at the tested point, so 100% means "verified at the tested parameters", not "proven everywhere".
• The reference is ForwardDiff through the same primal (including the internal h = 1e-8 finite difference in PrimaryCensored.logpdf, ~1e-6 floor), so "correct" means "agrees with ForwardDiff", not ground-truth analytic.

Ordering

This overlaps #252: AD coverage is almost entirely ext/ coverage, the exact attribution-across-Julia-versions problem raised there. The per-backend coverage jobs should each be pinned to one Julia version and walk ext/ explicitly, the same fix, so the two are worth doing together.

Separable from the in-flight Constant-context AD-fixtures change (#284); will land as its own PR once that is ready for review.

This was opened by a bot. Please ping @seabbs for any questions.

[seabbs]

Nice this looks like it works well. Just waiting on CI