Skip to content

refactor(proto): migrate sort merge join serde#23712

Merged
adriangb merged 2 commits into
apache:mainfrom
Phoenix500526:issue/23508
Jul 21, 2026
Merged

refactor(proto): migrate sort merge join serde#23712
adriangb merged 2 commits into
apache:mainfrom
Phoenix500526:issue/23508

Conversation

@Phoenix500526

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

SortMergeJoinExec serialization currently depends on centralized
encoding and decoding logic in datafusion-proto. This keeps protobuf
ownership separate from the execution plan and contributes to the
central type-dispatch chain tracked by #23494.

Moving serialization into SortMergeJoinExec keeps the implementation
with the plan that owns the relevant state and aligns it with the new
per-plan serialization model.

What changes are included in this PR?

  • Add ExecutionPlan::try_to_proto for SortMergeJoinExec.
  • Add SortMergeJoinExec::try_from_proto.
  • Preserve serialization of:
    • Both child plans.
    • Join-key expressions.
    • Join type and null-equality behavior.
    • Optional join filters and column origins.
    • Per-key sort options.
  • Convert JoinType, JoinSide, and NullEquality through exhaustive
    by-name matches.
  • Route decoding through the plan-local implementation and remove the
    centralized encoding downcast.
  • Retain the old public conversion helpers as deprecated compatibility
    APIs.
  • Preserve the existing protobuf wire representation.
  • Expand roundtrip coverage for every join type, both null-equality
    modes, filters, column origins, and non-default sort options.

Are these changes tested?

Yes

Are there any user-facing changes?

There are no runtime behavior or protobuf wire-format changes.

The old PhysicalPlanNodeExt conversion helpers remain available but
are deprecated in favor of the plan-local serialization hooks.

SortMergeJoin serialization still depends on centralized plan
downcasts, keeping protobuf ownership separate from the execution plan.

Move encoding and decoding behind its ExecutionPlan hooks while
preserving existing wire fields. Keep the legacy helpers as deprecated
compatibility APIs and verify join enums and sort options roundtrip.

Closes apache#23508

Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
@github-actions github-actions Bot added proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 20, 2026
@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  36.099s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.134s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  35.185s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.135s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.780s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  73.818s] datafusion-physical-plan
    Building datafusion-proto v54.0.0 (current)
       Built [  57.109s] (current)
     Parsing datafusion-proto v54.0.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v54.0.0 (baseline)
       Built [  57.086s] (baseline)
     Parsing datafusion-proto v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.249s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---

Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_marked_deprecated.ron

Failed in:
  method try_into_sort_join in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:719
  method try_from_sort_merge_join_exec in trait datafusion_proto::physical_plan::PhysicalPlanNodeExt in /home/runner/work/datafusion/datafusion/datafusion/proto/src/physical_plan/mod.rs:719

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [ 115.848s] datafusion-proto

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 20, 2026
@codecov-commenter

codecov-commenter commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.46479% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.70%. Comparing base (a8d1af6) to head (54ad24a).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
...on/physical-plan/src/joins/sort_merge_join/exec.rs 83.50% 16 Missing and 16 partials ⚠️
datafusion/proto/src/physical_plan/mod.rs 15.78% 16 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #23712    +/-   ##
========================================
  Coverage   80.69%   80.70%            
========================================
  Files        1088     1089     +1     
  Lines      367741   368329   +588     
  Branches   367741   368329   +588     
========================================
+ Hits       296761   297258   +497     
- Misses      53272    53355    +83     
- Partials    17708    17716     +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

protobuf::GenerateSeriesName::GsRange => "range",
}
}
#[deprecated(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have these deprecated methods delegate to the new methods so we avoid code duplication?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

SortMergeJoinExec serde now lives in per-plan hooks, but the deprecated
PhysicalPlanNodeExt helpers kept independent implementations.
Duplicating join, filter, and sort-option conversions allowed the
compatibility path to drift from normal dispatch.

Build the existing encode and decode adapter contexts in those helpers
and forward to SortMergeJoinExec's canonical hooks. Remove the protobuf
sort-expression alias that the duplicate encoder no longer needs.

Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@adriangb
adriangb added this pull request to the merge queue Jul 21, 2026
Merged via the queue into apache:main with commit 9a50c67 Jul 21, 2026
40 checks passed
@Phoenix500526
Phoenix500526 deleted the issue/23508 branch July 21, 2026 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proto: migrate SortMergeJoinExec

3 participants