Skip to content

feat: complete range repartition physical planning#23617

Open
saadtajwar wants to merge 8 commits into
apache:mainfrom
saadtajwar:saadtajwar/finish-range-repartition-physical-plan
Open

feat: complete range repartition physical planning#23617
saadtajwar wants to merge 8 commits into
apache:mainfrom
saadtajwar:saadtajwar/finish-range-repartition-physical-plan

Conversation

@saadtajwar

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

After #23231 was merged in for supporting physical execution of the range repartitioning scheme, we still had a few methods on physical planning unimplemented, specifically try_swapping_with_projection, try_pushdown_sort, repartitioned - this PR finishes the implementation of those methods

What changes are included in this PR?

  • try_swapping_with_projection: similar to the Hash scheme, for Range we call update_expr for each of the range key expressions to attempt rewriting based on the projection expressions
  • try_pushdown_sort: same as other variants, we delegate to the child and wrap with a new RepartitionExec
  • repartitioned: unable to support for Range, left comment in codebase with explanation

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 15, 2026
@saadtajwar
saadtajwar marked this pull request as ready for review July 15, 2026 16:15
@saadtajwar

Copy link
Copy Markdown
Contributor Author

cc @gene-bordegaray & @gabotechs ! Thanks in advance for any feedback! 😁 🎉

@gene-bordegaray gene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some feedback, thank you 🙇

Comment thread datafusion/physical-plan/src/repartition/mod.rs Outdated
Comment thread datafusion/physical-plan/src/repartition/mod.rs
Comment thread datafusion/physical-plan/src/repartition/mod.rs
Comment thread datafusion/physical-plan/src/repartition/mod.rs Outdated
@saadtajwar

Copy link
Copy Markdown
Contributor Author

@gene-bordegaray thanks for the feedback! Pushed a commit to address - appreciate all the time taken reviewing!

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.18121% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.65%. Comparing base (e104138) to head (8a544e6).

Files with missing lines Patch % Lines
datafusion/physical-plan/src/repartition/mod.rs 77.18% 35 Missing and 33 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #23617    +/-   ##
========================================
  Coverage   80.65%   80.65%            
========================================
  Files        1086     1086            
  Lines      366521   366814   +293     
  Branches   366521   366814   +293     
========================================
+ Hits       295618   295858   +240     
- Misses      53274    53296    +22     
- Partials    17629    17660    +31     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gene-bordegaray gene-bordegaray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating this, I think my earlier comment may have pushed this a little too far in the helper direction 😅

What I was trying to get at was mostly reducing the repeated range-partitioning setup, not adding a bunch of extra test scaffolding / coverage. I think we can keep this much tighter:

  • one positive projection-swap test that verifies the range key is rewritten when its projected column index changes
  • one negative projection-swap test that covers dropping a single key and dropping part of a compound key
  • two sort-pushdown tests: supported when the range repartition maintains input order, unsupported when it does not
  • keep the repartitioned() None test

I’d drop the extra execution sortedness test since this PR really about the planning behavior.

For helpers I think the useful ones are only the small range-specific setup things like build range partitioning from cols, build a projection from cols, unwrap/assert range partitioning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complete physical execution planning of range partitioning

3 participants