Skip to content

Optimize redundant calls to UDFs producing StructArray #23655

Description

@liningpan

Is your feature request related to a problem or challenge?

Since UDF can only return a single column as output, returning a StructArray is desired in many cases. However, accessing fields of the returned struct results in redundant calls to the UDF.

SELECT
    example_struct(c)['a'],
    example_struct(c)['b']
FROM (SELECT unnest(range(1,100)) as c)

Datafusion produces the following projection step for the query, resulting in calling the UDF with the same column twice.

ProjectionExec: expr=[get_field(example_struct(c@0), a) as example_struct(c)[a], get_field(example_struct(c@0), b) as example_struct(c)[b]]

Note that calling the UDF in a CTE or view does not change the execution plan.

Here is a minimal example https://github.com/liningpan/udf-get-field-optimization.

Describe the solution you'd like

@adriangb mentioned that CSE (Common Sub-expression Elimination) optimizer might be able to handle this.

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Fields

No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions