Commit d91dcb7
authored
feat(functions-nested): add array_filter higher-order function (apache#21895)
## Which issue does this PR close?
Partially addresses apache#14509 — implements `array_filter` / `list_filter`.
## Rationale for this change
`array_transform` (apache#21679) added the first `HigherOrderUDF`.
`array_filter` is the natural companion: filter array elements with a
boolean lambda, matching Spark `filter` / DuckDB `list_filter`
semantics.
## What changes are included in this PR?
- New `HigherOrderUDF` `ArrayFilter` (`array_filter` / `list_filter`
alias)
- Boolean lambda per element; `true` keeps, `false`/null drops (matches
Spark semantics)
- Handles `List`, `LargeList`, sliced arrays, null sublists
- Scalar predicate short-circuit (`x -> true` / `x -> false`)
- No-copy fast path when nothing is filtered (skips
`arrow::compute::filter`)
- Shared HOF helpers extracted from `array_transform` into a common
module (`value_lambda_pair`, `coerce_single_list_arg`,
`single_list_lambda_parameters`, `extract_list_values`)
- Shared unit test helpers for higher-order function tests
## Are these changes tested?
- Unit tests: basic filter, multiple sublists, sliced arrays, null
sublists, all-filtered-out, nothing-filtered (fast path), scalar
true/false predicates
- SQL logic tests in `array_filter.slt`: filter variants, `array_filter`
+ `array_transform` combinations, error cases
## Are there any user-facing changes?
Yes — `array_filter(array, lambda)` and alias `list_filter(array,
lambda)` are now available as SQL functions.1 parent 707d861 commit d91dcb7
6 files changed
Lines changed: 937 additions & 119 deletions
File tree
- datafusion
- functions-nested/src
- sqllogictest/test_files/array
- docs/source/user-guide/sql
0 commit comments