Skip to content

feat: add DataFrame.writeCsv with CsvWriteOptions #38

@andygrove

Description

@andygrove

Is your feature request related to a problem or challenge?

DataFrame.writeParquet shipped in #27, but the symmetric CSV writer is
missing. DataFusion exposes DataFrame::write_csv, which is commonly
used for ad-hoc exports.

Describe the solution you'd like

  • Add a CsvWriteOptions value class covering the knobs that
    DataFrameWriteOptions + CSV-specific options expose
    (single-file output, partition columns, header, delimiter, quote,
    escape, null-value string, compression).
  • Add proto/csv_write_options.proto, following the convention
    established by parquet_write_options.proto in feat(write): add DataFrame.writeParquet with ParquetWriteOptions #27.
  • Expose DataFrame.writeCsv(path[, options]).
  • Cover with tests in the spirit of DataFrameWriteParquetTest
    (single-file, compression, retain-after-write semantics).

Describe alternatives you've considered

COPY (SELECT * FROM df) TO '…' via SQL works but requires a
registered name and a SQL detour.

Additional context

Aligns the writer story: with this and the JSON writer, we cover the
three formats DataFusion's DataFrame API can write today
(write_parquet, write_csv, write_json).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions