Skip full-grid k0 memory term for dask kriging templates#2926
Merged
brendancol merged 1 commit intoJun 4, 2026
Conversation
The kriging() memory guard estimated the prediction matrix (k0) from the full grid size, even for dask-backed templates where map_blocks builds k0 one chunk at a time. A large chunked template raised a spurious MemoryError for the case the dask backend exists to handle. Drop the k0 term when the template is dask-backed; the point-based variogram-pair and matrix terms still apply. Also records the interpolate-kriging performance sweep state.
brendancol
commented
Jun 4, 2026
brendancol
left a comment
Contributor
Author
There was a problem hiding this comment.
PR Review: Skip full-grid k0 memory term for dask kriging templates
Blockers
None.
Suggestions
None.
Nits
_kriging.py_check_kriging_memory: withis_dask=Truethe k0 term is dropped unconditionally. A dask array with a single chunk covering the whole grid still allocates the full k0 in that one chunk, so the guard under-estimates that case. It is unusual to hold a full-grid dask template in one chunk, and the old full-grid estimate was wrong for the normal chunked case, so dropping the term is the right tradeoff here. If you want to tighten it later, scale k0 to the largest chunk viatemplate.data.chunks.
What looks good
- The change is small: one keyword arg and a per-backend choice for a single term. The variogram-pair and matrix terms still apply on the dask path, which is correct because both are built on the host regardless of backend.
is_daskdetection checksda is not Nonebefore the isinstance, so it is safe with dask uninstalled, and it covers dask+cupy since that backend also wraps adask.array.Array.- Tests cover the helper (k0 skipped, matrix still guarded), an end-to-end 4000x4000 / 200x200 dask template under a 64 MB cap, and the unchanged numpy path.
Checklist
- Algorithm matches reference: n/a (memory guard, not a numerical kernel)
- All implemented backends consistent: yes, dask+numpy and dask+cupy both covered
- NaN handling: unchanged
- Edge cases covered by tests: yes
- Dask chunk boundaries: n/a
- No premature materialization: confirmed, no .values/.compute added
- Benchmark: not needed (guard logic)
- README matrix: n/a (no new function)
- Docstrings: updated to document is_dask behavior
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2923
What changed
_check_kriging_memorynow takes anis_daskflag. When the template is dask-backed, it drops the prediction-matrix (k0) term from the estimate, sincemap_blocksbuildsk0one chunk at a time and peak per-task memory scales with the chunk, not the full grid.kriging()detects a dask-backed template viaisinstance(template.data, da.Array)and passes the flag through.Why
Before this, a large chunked dask template raised a spurious
MemoryErrorfor the exact case the dask backend exists to handle. A 4000x4000 template with 200x200 chunks claimed it needed ~8.1 GB when each chunk only needs ~20 MB.Backends
numpy guard behaviour unchanged. dask+numpy and dask+cupy no longer hit the full-grid k0 estimate. cupy (non-dask) unchanged.
Test plan
test_dask_template_skips_grid_memory_guard: a 4000x4000 / 200x200 dask template runs under a 64 MB captest_check_helper_dask_skips_k0_term: the helper drops the k0 term whenis_dask=Truetest_check_helper_dask_still_guards_matrix: the point-based matrix guard still fires on the dask pathtest_interpolation.pygreen (44 passed, includes cupy and dask+cupy parity)