You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge?
DataFusion 53.1 supports Arrow IPC files via SessionContext::read_arrow / register_arrow. Since results already
return to the JVM as Arrow batches over the C Data Interface, being
able to round-trip through Arrow IPC files is a natural fit and the
cheapest format to integrate.
Describe the solution you'd like
Add an ArrowReadOptions value class (file extension, schema).
Is your feature request related to a problem or challenge?
DataFusion 53.1 supports Arrow IPC files via
SessionContext::read_arrow/register_arrow. Since results alreadyreturn to the JVM as Arrow batches over the C Data Interface, being
able to round-trip through Arrow IPC files is a natural fit and the
cheapest format to integrate.
Describe the solution you'd like
ArrowReadOptionsvalue class (file extension, schema).proto/arrow_read_options.protoand pass options through theproto-over-JNI convention (refactor: pass csv and parquet read options via protobuf #29).
SessionContext.registerArrow(name, path[, options])andreadArrow(path[, options]).writer to keep test deps minimal.
Describe alternatives you've considered
CREATE EXTERNAL TABLE … STORED AS ARROWvia SQL.Additional context
In the default DataFusion feature set; no Cargo flag changes required.