Is your feature request related to a problem or challenge?
I would like to be able to register custom data sources implemented in Java. This has already been implemented in Comet so we can learn from the approach there.
We would need a new trait in Java to represent a data source, or possibly just implement an Iterator type. We need to extend the protobuf to be able to pass this custom source into native code. Native code would then need a data source implementation that makes JNI calls into the JVM to fetch batches.
In Comet, see ScanExec in native code, and CometBatchIterator in Scala code.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Is your feature request related to a problem or challenge?
I would like to be able to register custom data sources implemented in Java. This has already been implemented in Comet so we can learn from the approach there.
We would need a new trait in Java to represent a data source, or possibly just implement an Iterator type. We need to extend the protobuf to be able to pass this custom source into native code. Native code would then need a data source implementation that makes JNI calls into the JVM to fetch batches.
In Comet, see
ScanExecin native code, andCometBatchIteratorin Scala code.Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response