Skip to content

feat(dataframe): expose withColumn and unnestColumns #41

@andygrove

Description

@andygrove

Is your feature request related to a problem or challenge?

The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.

Describe the solution you'd like

  • DataFrame.withColumn(String name, String expr). The expression is
    a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
    convention already used by DataFrame.filter(String). Behaviour
    mirrors DataFusion::DataFrame::with_column — replaces a column of
    the same name in place, otherwise appends.
  • DataFrame.unnestColumns(String... columns) and an overload
    accepting an UnnestOptions value (preserve-nulls, recursions). Maps
    to unnest_columns / unnest_columns_with_options.
  • Tests in DataFrameTransformationsTest (round-trip on a small
    fixture, schema check, error on unknown column).

Describe alternatives you've considered

SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.

Additional context

Follows the same JNI shape as #19's filter(String)parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions