Skip to content

README.md

Latest commit

 

History

History
256 lines (213 loc) · 9.72 KB

README.md

File metadata and controls

256 lines (213 loc) · 9.72 KB
Raw

Converter Task

The converter task converts data between different formats, supporting CSV, HTML, XLSX, XLS, EML (Email), Protobuf, and other data format transformations.

Function

The converter task transforms data from one format to another, enabling interoperability between different data formats and systems.

Behavior

The converter task transforms data between different formats. It receives records from its input channel, converts the data from the source format to the target format using the specified options, and sends the converted records to its output channel.

  • If skip_first is True and no columns are provided, then the column names in the output will be the values in the first row of the CSV file, normalized: non-alphanumeric characters are replaced by underscores, leading/trailing underscores are trimmed, and the result is lowercased.
  • If skip_first is True and columns are provided, then the column names in the output will be the values provided (i.e., Provided column names supersede names from first row).
  • If skip_first is False, and no columns are provided, then the column names in the output will be named col1, col2, col3, etc.
  • If skip_first is False, and columns are provided, then the column names in the output will be the values provided.
  • A leading UTF-8 BOM is stripped from the first record, so it neither breaks parsing nor leaks into a column name or value.

Configuration Fields

Field Type Default Description
name string - Task name for identification
type string converter Must be "converter"
format string - Format to convert to (csv, html, sst, xlsx, xls, eml, protobuf)
delimiter string - SST only: separator between key and value

CSV Format Options

Field Type Default Description
skip_first bool false Skip the first record (useful for headers)
columns array - Array of column definitions
columns[].name string - Name for the column
columns[].is_numeric bool false Whether the column contains numeric data

HTML Format Options

Field Type Default Description
container string - XPath expression to select specific container elements

EML Format Options

The EML converter does not have specific configuration options. It automatically parses the email content and extracts:

  • HTML Body: Saved as body.html
  • Text Body: Saved as body.txt
  • Headers: Saved as headers.json (key-value pairs of all email headers)
  • Attachments: Saved with sanitized filenames — non-alphanumeric characters are replaced with underscores, leading/trailing underscores are trimmed, the result is lowercased, and names longer than 200 characters are truncated
  • Inline Images: Saved with sanitized filenames (same rules as attachments)

Metadata generated for each output:

  • converter_filename: The name of the output file
  • content_type: The MIME type of the content

Protobuf Format Options

Decodes binary protobuf payloads to JSON using a compiled FileDescriptorSet (produced by protoc --descriptor_set_out=foo.desc --include_imports foo.proto).

Field Type Default Description
descriptor_path string - Path to a binary FileDescriptorSet. Accepts a local filesystem path or an s3://bucket/key URI
message_name string - Fully-qualified message name (e.g. pkg.MyMessage)
region string us-west-2 AWS region used when descriptor_path is an s3:// URI. Ignored for local paths
use_proto_names bool false Emit field names as defined in .proto instead of lowerCamelCase
emit_unpopulated bool false Include zero-valued fields in output

The descriptor is fetched once per task instance (cached for the lifetime of the run); S3 credentials are resolved from the standard AWS SDK chain (env, profile, IRSA, EC2 IMDS).

Example — local descriptor:

tasks:
  - name: decode_event
    type: converter
    format: protobuf
    descriptor_path: schemas/events.desc
    message_name: events.v1.UserEvent

Example — descriptor in S3:

tasks:
  - name: decode_event
    type: converter
    format: protobuf
    descriptor_path: s3://my-bucket/schemas/events.desc
    message_name: events.v1.UserEvent
    region: us-east-1

SST Format Options

Convert a single line to the SSTable which could be stored on s3 or via file. It expects a single line as input

XLSX / XLS Format Options

Field Type Default Description
sheets array all sheets Optional array of sheet names to process. If not specified, all sheets are processed
skip_rows int 0 Number of rows to skip from the beginning of each sheet (e.g., for header rows)
skip_rows_by_sheet map[string]int - Per-sheet row skip overrides. Keys are sheet names, values are rows to skip. Takes precedence over skip_rows
sanitize_headers bool false If true, normalizes header row values: non-alphanumeric characters are replaced by underscores, leading/trailing underscores are trimmed, and the result is lowercased. Assumes the first unskipped row to be header
sanitize_sheet_names bool false If true, normalizes sheet names: non-alphanumeric characters are replaced by underscores, leading/trailing underscores are trimmed, and the result is lowercased before storing in the xlsx_sheet_name context key

Important: Both converters emit one record per sheet. Each record contains the sheet's data in CSV format, with the sheet name available in the record context under the key xlsx_sheet_name.

Supported Formats

The converter supports the following formats:

  • CSV: Converts CSV data to JSON with column mapping and type conversion
  • HTML: Converts HTML to JSON representation with element structure
  • XLSX: Converts modern Excel files to CSV format. Note: Each sheet is emitted as a separate record with the sheet name stored in the context (key: xlsx_sheet_name)
  • XLS: Converts legacy Excel 97-2003 files (.xls, BIFF8) to CSV format. Same options and per-sheet output as XLSX
  • EML: Converts EML (Email) files to their constituent parts (HTML body, Text body, Attachments)
  • Protobuf: Decodes binary protobuf messages to JSON using a compiled FileDescriptorSet

Example Configurations

CSV to JSON conversion:

tasks:
  - name: csv_to_json
    type: converter
    format: csv
    skip_first: true
    columns:
      - name: id
        is_numeric: true
      - name: name
      - name: email
      - name: age
        is_numeric: true

HTML to JSON conversion:

tasks:
  - name: html_to_json
    type: converter
    format: html
    container: "//div[@class='content']"

EML processing:

tasks:
  - name: read_email
    type: file
    path: test_email
  - name: process_email
    type: converter
    format: eml
  # Output will correspond to body parts and attachments

Excel to CSV conversion (all sheets):

tasks:
  - name: read_excel
    type: file
    path: data.xlsx
  - name: convert_excel
    type: converter
    format: xlsx
  - name: echo
    type: echo
    only_data: true

Legacy Excel 97-2003 (.xls) to CSV:

tasks:
  - name: read_excel
    type: file
    path: data.xls
  - name: convert_excel
    type: converter
    format: xls
  - name: echo
    type: echo
    only_data: true

Excel to CSV conversion (specific sheets):

tasks:
  - name: read_excel
    type: file
    path: report.xlsx
  - name: convert_excel
    type: converter
    format: xlsx
    sheets: ["Sales", "Inventory"]
  - name: echo_sheet_name
    type: echo
    # Each record will have xlsx_sheet_name in context

Excel to CSV with sanitized headers:

tasks:
  - name: read_excel
    type: file
    path: report.xlsx
  - name: convert_excel
    type: converter
    format: xlsx
    sanitize_headers: true  # "First Name" becomes "first_name", "Sales (USD)" becomes "sales_usd"

Excel to CSV with sanitized sheet names:

tasks:
  - name: read_excel
    type: file
    path: report.xlsx
  - name: convert_excel
    type: converter
    format: xlsx
    sanitize_sheet_names: true  # "Sales Q1" becomes "sales_q1" in context key xlsx_sheet_name

Excel to CSV with row skipping:

tasks:
  - name: read_excel
    type: file
    path: report.xlsx
  - name: convert_excel
    type: converter
    format: xlsx
    skip_rows: 1  # Skip header row on all sheets
    skip_rows_by_sheet:
      Summary: 3  # Skip 3 rows on Summary sheet (overrides skip_rows)
      RawData: 0  # Don't skip any rows on RawData sheet

Sample Pipelines

  • test/pipelines/convert_file.yaml - File format conversion
  • test/pipelines/convert_industries.yaml - Data format transformation
  • test/pipelines/converter/convert_xls.yaml - Excel to CSV conversion
  • test/pipelines/converter/eml.yaml - MIME/EML email parsing
  • test/pipelines/converter/protobuf.yaml - Protobuf decoding

Use Cases

  • Data format conversion: Convert between different data formats
  • Email processing: Parse MIME/EML files to extract bodies and attachments
  • Excel processing: Extract and process data from Excel spreadsheets, with separate handling for each sheet
  • API integration: Transform data for different API requirements
  • Database migration: Convert data for different database systems
  • Report generation: Convert data to report-friendly formats
  • Data exchange: Enable data sharing between different systems
  • ETL workflows: Transform data as part of extract, transform, load processes