Feature: Implement zero-copy conversion from C++ ArFrame to Apache Arrow RecordBatch

[knoxiboy]

Problem or motivation

Currently, handing off data from Arnio's C++ core to Python (e.g., Pandas) likely incurs expensive memory copying or serialization overhead. For large datasets (10GB+), this handoff is a severe bottleneck.

Proposed solution

Implement a zero-copy API that exports the internal C++ ArFrame directly to an Apache Arrow RecordBatch or Table using the Arrow C Data Interface.

User story

As a data engineer processing massive CSV files, I want Arnio to parse and clean the data in C++, and then pass it to Pandas/Polars via PyArrow without duplicating the data in RAM, so that I don't hit Out-Of-Memory (OOM) errors.

Acceptance criteria

• C++ core implements the Arrow C Data Interface structure.
• Python bindings expose a .to_arrow() method.
• Unit tests verify zero-copy behavior and data integrity.

Alternatives considered

Exporting to Parquet to disk and reading back in, but disk I/O defeats the purpose of an in-memory pipeline.

Scope

• pandas interoperability
• Performance optimization
• New pipeline step (C++)

Expected difficulty

• Advanced

Additional context

References: Apache Arrow C Data Interface documentation.

[knoxiboy]

/assign gssoc'26
Hi! @im-anishraj
I would like to work on this issue under GSSoC'26. Please assign this issue to me.

[im-anishraj]

@knoxiboy thanks for your interest. I am not assigning this yet because the issue is still status:needs-triage; the zero-copy Arrow scope needs maintainer decisions on API shape, ownership/lifetime guarantees, dependency/build impact, and test expectations before contributor work starts. Please wait until it is moved to status:ready, then pick one issue and post a concrete implementation plan before assignment.