fix: Support RANGE window frames over binary ORDER BY keys - #24357
Conversation
ce39143 to
5951847
Compare
cbcb22f to
e0663ae
Compare
Closes apache#24327. A window with an `ORDER BY` over a binary column and no explicit frame gets the default `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. Computing that frame's bounds goes through `extract_window_frame_target_type`, which had no arm for the binary types, so any window function whose leading `ORDER BY` key was `Binary`/`LargeBinary`/`BinaryView`/`FixedSizeBinary` failed with: Internal error: Cannot run range queries on datatype: Binary Binary is orderable, so peer/range comparison is well defined for it in exactly the same way it already is for `Utf8`. Accept the binary types alongside the string types. A finite RANGE offset such as `1 PRECEDING` asks for more than ordering: the bound is computed as `current_value - 1`, so the order key type must also support arithmetic. Binary does not, and neither do the `Utf8`, `Boolean`, `List` and `Null` order keys that `extract_window_frame_target_type` already accepted. For those, `ScalarValue::sub_checked` failed during execution and the error was swallowed by the overflow handling from apache#22140, which collapses a failed bound to the partition edge. So SELECT COUNT(*) OVER (ORDER BY x RANGE BETWEEN 1 PRECEDING AND CURRENT ROW) FROM (VALUES ('a'), ('b'), ('c')) t(x) silently returned the running count 1, 2, 3 -- the whole partition -- rather than an error. Reject the offset form during planning for every target type that has no arithmetic. For the string, boolean, list and null order keys this turns a wrong result into a plan error, which matches PostgreSQL's "RANGE with offset PRECEDING/FOLLOWING is not supported for column type ...". `WindowFrame::free_range()` becomes public so that check can be expressed in the vocabulary the crate already uses for "bounds that need no arithmetic". Document the restriction in the window functions user guide, and the changed SQL behavior in the 55.0.0 upgrade guide. Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
e0663ae to
146f516
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24357 +/- ##
==========================================
+ Coverage 81.19% 81.26% +0.06%
==========================================
Files 1110 1112 +2
Lines 388616 391585 +2969
Branches 388616 391585 +2969
==========================================
+ Hits 315527 318203 +2676
- Misses 54509 54675 +166
- Partials 18580 18707 +127 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kosiew
left a comment
There was a problem hiding this comment.
@fornwall, thanks for the follow-up. The added Dictionary(Int32, Binary) coverage in window.slt exercises RANGE BETWEEN 1 PRECEDING AND CURRENT ROW and confirms that the dictionary type is unwrapped before the finite-offset arithmetic rejection. That covers the case I was looking for.
I also reran the relevant validation and didn't find any new issues. Looks good to me, thanks!
|
🚀 |
|
The PR was removed from the queue. But that seems like some general CI issue, not specific to this PR: https://github.com/apache/datafusion/actions/runs/32227729624/job/95990834191 |
Which issue does this PR close?
ORDER BYuses a binary column #24327.Rationale for this change
A window function over a binary
ORDER BYkey fails even with no frame clause:ORDER BYwithout a frame defaults toRANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, andextract_window_frame_target_typehad no arm for the binary types.LargeBinary,BinaryView,FixedSizeBinaryand dictionary-wrapped binary are affected too. Those bounds only need to compare order key values, which binary supports just asUtf8does.A finite offset like
1 PRECEDINGis different: the bound is computed ascurrent_value - 1, which needs arithmetic on the order key type. Binary has none, and neither doUtf8,Boolean,ListandNull-- yet offset frames overUtf8,BooleanandListkeys were accepted, and since 54.0.0 the failing arithmetic is swallowed by the overflow handling from #22140, silently collapsing the bound to the partition edge:returns the running count
1,2,3-- a silently widened frame -- instead of an error. This PR rejects such frames at planning time instead, matching PostgreSQL.What changes are included in this PR?
extract_window_frame_target_type. The existing dictionary arm already recurses into the value type.Interval. The message follows PostgreSQL'sRANGE with offset PRECEDING/FOLLOWING is not supported for column type ....WindowFrame::free_range()pub, so the check reuses the crate's existing notion of "bounds that are UNBOUNDED or CURRENT ROW".55.0.0upgrade guide.Are these changes tested?
Yes, in
window.slt: all four binary types plus dictionary-wrapped binary, ascending and descending, an aggregate andRANK, with duplicate values asserting peer semantics rather than positional counting. Planning errors for offsets overBinary,Utf8,Boolean,List(Int64)andNull, on the end bound as well as the start. Free range frames overUtf8andBooleanstill return results, andDecimal128keeps its offsets.The full sqllogictest suite passes.
Are there any user-facing changes?
Yes, one breaking SQL change:
RANGE BETWEEN <offset> PRECEDING/FOLLOWINGover aUtf8,Binary,Boolean,ListorNullORDER BYkey now fails at planning time withRANGE with offset PRECEDING/FOLLOWING is not supported for ORDER BY type .... OverUtf8,BooleanandListkeys this replaces the silently widened frames described above; overNulland binary keys such queries already failed at planning time, so only the error message changes. The55.0.0upgrade guide entry covers the per-type history and the migration paths: use aROWSframe to count rows, or stateUNBOUNDED PRECEDING/UNBOUNDED FOLLOWINGexplicitly to compare against the partition edge.In the other direction, binary order keys now work with free range frames, where they previously raised the internal error above.
WindowFrame::free_range()going from private topubis additive, so not a breaking Rust API change.AI usage: Created with Claude Code and Opus 5. I have reviewed the code and made modifications where it made sense. I have also tested this end to end in an internal project.