Is your feature request related to a problem or challenge?
The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.
Describe the solution you'd like
DataFrame.withColumn(String name, String expr). The expression is
a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
convention already used by DataFrame.filter(String). Behaviour
mirrors DataFusion::DataFrame::with_column — replaces a column of
the same name in place, otherwise appends.
DataFrame.unnestColumns(String... columns) and an overload
accepting an UnnestOptions value (preserve-nulls, recursions). Maps
to unnest_columns / unnest_columns_with_options.
- Tests in
DataFrameTransformationsTest (round-trip on a small
fixture, schema check, error on unknown column).
Describe alternatives you've considered
SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.
Additional context
Follows the same JNI shape as #19's filter(String) — parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.
Is your feature request related to a problem or challenge?
The DataFrame API gained
dropColumnsandwithColumnRenamedin #30but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing.
unnestColumnsforstruct / list flattening is in the same family.
Describe the solution you'd like
DataFrame.withColumn(String name, String expr). The expression isa SQL fragment parsed via DataFusion's
parse_sql_expr, matching theconvention already used by
DataFrame.filter(String). Behaviourmirrors
DataFusion::DataFrame::with_column— replaces a column ofthe same name in place, otherwise appends.
DataFrame.unnestColumns(String... columns)and an overloadaccepting an
UnnestOptionsvalue (preserve-nulls, recursions). Mapsto
unnest_columns/unnest_columns_with_options.DataFrameTransformationsTest(round-trip on a smallfixture, schema check, error on unknown column).
Describe alternatives you've considered
SELECT a + b AS c, *viactx.sql(...). Works forwithColumn, butrequires the DataFrame to be registered as a table first; awkward for
intermediate transformations.
Additional context
Follows the same JNI shape as #19's
filter(String)—parse_sql_expron the native side, no Java-side
Exprmodel required. Keeps thisissue independent of #N (joins issue) where an
Exprbuilder mayeventually be needed.