Skip to content

Aggregate dynamic filter can prune rows required by unsupported expressions #24816

Description

@lyne7-sc

Describe the bug

When an AggregateExec contains both supported aggregate expressions such as MIN(a) and an unsupported expression such as MIN(c + 1), DataFusion builds a dynamic filter using only the supported expressions.

Because this filter is applied to the shared input scan, it can prune rows that are still required by the unsupported aggregate and produce incorrect results.

To Reproduce

Use two Parquet files containing:

file_0: (1, 12, 100), (8, 4, 70)
file_1: (1, 6, 90), (8, 12, 110)

Then run:

SELECT MIN(a), MAX(a), MAX(b), MIN(c + 1)
FROM t;

The expected result for MIN(c + 1) is 71.

Currently, the supported aggregates can produce the following dynamic filter:

a < 1 OR a > 8 OR b > 12

This filter does not account for MIN(c + 1). Depending on execution order, it can prune file_0 and produce 91 instead of 71.

Expected behavior

Aggregate dynamic filtering should only be enabled when a safe predicate can be produced for every aggregate expression.

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions