XTable REST Service for metadata and catalog sync.

[vinishjail97]

Feature Request / Improvement

We have support to run XTable sync in continuous mode after this PR was merged #593

There was interest in the community to build a REST service, users can call to trigger the metadata/catalog sync without the need for installing/packaging the dependencies in their project. Implementation details are still not finalized and feature owner can take it up.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[vaibhavk1992]

I am willing to take this up.

[vaibhavk1992]

Hi team

Can we please add what all API end points we are planning to have in this and the consideration to take. I have listed a few though.

Trigger runsync Sync
Endpoint: POST /api/v1/sync/runsync​
Description: Initiates the synchronization of metadata across supported table formats.​
Request Body: May include parameters specifying the scope or type of metadata to sync.​

Endpoint: GET /api/v1/sync/status​
Description: Retrieves the current status of ongoing or recent synchronization processes.​
Query Parameters: Could include filters for specific sync operations or time ranges.​
List Supported Table Formats

Endpoint: GET /api/v1/formats​
Description: Returns a list of table formats supported by the synchronization service.

[sapienza88]

Will this PR be useful for the Polaris integration? I am curious to see the full list of APIs as well.

[vinishjail97]

@unical1988 Yes that's correct.

[amoghnatu]

@vinishjail97 Have you checked this? https://aws.amazon.com/blogs/big-data/run-apache-xtable-in-aws-lambda-for-background-conversion-of-open-table-formats/

That should give you an endpoint to invoke to perform the conversion. You can do the same on Azure Functions.