Describe the bug
When an AggregateExec contains both supported aggregate expressions such as MIN(a) and an unsupported expression such as MIN(c + 1), DataFusion builds a dynamic filter using only the supported expressions.
Because this filter is applied to the shared input scan, it can prune rows that are still required by the unsupported aggregate and produce incorrect results.
To Reproduce
Use two Parquet files containing:
file_0: (1, 12, 100), (8, 4, 70)
file_1: (1, 6, 90), (8, 12, 110)
Then run:
SELECT MIN(a), MAX(a), MAX(b), MIN(c + 1)
FROM t;
The expected result for MIN(c + 1) is 71.
Currently, the supported aggregates can produce the following dynamic filter:
a < 1 OR a > 8 OR b > 12
This filter does not account for MIN(c + 1). Depending on execution order, it can prune file_0 and produce 91 instead of 71.
Expected behavior
Aggregate dynamic filtering should only be enabled when a safe predicate can be produced for every aggregate expression.
Additional context
No response
Describe the bug
When an
AggregateExeccontains both supported aggregate expressions such asMIN(a)and an unsupported expression such asMIN(c + 1), DataFusion builds a dynamic filter using only the supported expressions.Because this filter is applied to the shared input scan, it can prune rows that are still required by the unsupported aggregate and produce incorrect results.
To Reproduce
Use two Parquet files containing:
Then run:
The expected result for
MIN(c + 1)is 71.Currently, the supported aggregates can produce the following dynamic filter:
a < 1 OR a > 8 OR b > 12This filter does not account for
MIN(c + 1). Depending on execution order, it can prune file_0 and produce 91 instead of 71.Expected behavior
Aggregate dynamic filtering should only be enabled when a safe predicate can be produced for every aggregate expression.
Additional context
No response