feat(arrow): resolve S3-compatible schemes in ArrowFileSystemFileIO by plusplusjiajia · Pull Request #703 · apache/iceberg-cpp

plusplusjiajia · 2026-06-07T10:17:28Z

ArrowFileSystemFileIO::ResolvePath passed the full URI to the wrapped FileSystem's PathFromUri, which only accepts that FileSystem's native scheme(s3:// for S3FileSystem). S3-compatible object stores are commonly addressed with other schemes — s3a/s3n, or vendor schemes such as gs:// (GCS) andoss:// (Alibaba OSS) — served by an arrow::fs::S3FileSystem configured with an endpoint_override. Every read and write of such a location failed with:

expected a URI with one of the schemes (s3) but received <scheme>://...

Copilot

Pull request overview

This PR updates iceberg::arrow::ArrowFileSystemFileIO::ResolvePath to better handle S3-compatible (and other “foreign”) URI schemes by resolving them to the underlying Arrow filesystem path, avoiding failures when the wrapped Arrow FileSystem only accepts its native scheme.

Changes:

Adjusted ArrowFileSystemFileIO::ResolvePath to fall back from PathFromUri() to a scheme-less path when URI parsing fails.
Added a regression test ensuring reads succeed when a non-native scheme is used with the underlying filesystem.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/iceberg/arrow/arrow_io.cc	Implements URI scheme fallback behavior in `ResolvePath`.
src/iceberg/test/arrow_io_test.cc	Adds test coverage for resolving a foreign scheme to the underlying path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

plusplusjiajia · 2026-06-08T08:02:42Z

+  if (auto pos = file_location.find("://"); pos != std::string::npos) {
+    auto path = arrow_fs_->PathFromUri(file_location);
+    if (path.ok()) {
+      return path.ValueOrDie();
+    }
+    // PathFromUri rejects S3-compatible schemes (s3a/s3n, gs://, oss://);
+    // fall back to the scheme-less bucket/key.
+    return file_location.substr(pos + 3);
  }


Addressed: the fallback now triggers only on Arrow's scheme-mismatch error; other PathFromUri failures (malformed URI, unsupported authority, …) propagate. ?query/#fragment are stripped. Added tests for the propagate path and for preserving percent-encoded keys.

MisterRaindrop · 2026-06-08T07:25:47Z

+    }
+    // PathFromUri rejects S3-compatible schemes (s3a/s3n, gs://, oss://);
+    // fall back to the scheme-less bucket/key.
+    return file_location.substr(pos + 3);


I think this fallback is too broad. As implemented, any URI scheme rejected by PathFromUri will be stri
pped and interpreted as a path in the wrapped filesystem. For example, with an S3FileSystem, a mismatch ed URI such as file://bucket/prefix would be interpreted as bucket/prefix instead of failing fast.

Can we make the fallback explicit and narrow?

The scheme should be in an allowlist for the wrapped filesystem. For S3FileSystem, I think we can
support well-known aliases such as s3a and s3n. Vendor schemes such as oss or gs should probably be opt- in or discussed separately, because they may also represent native OSS/GCS locations.

The PathFromUri failure should match the expected scheme-mismatch / unsupported-scheme case. Other errors, such as malformed URI, authority handling errors, or local-path errors, should still be returned to the caller.

In other words, this should be guarded by both a scheme allowlist and the expected error condition,
rather than falling back for every PathFromUri failure.

@MisterRaindrop Agreed — the scheme allowlist belongs at the selection layer (like Java's ResolvingFileIO.SCHEME_TO_FILE_IO), so DetectBuiltinFileIO now accepts s3a/s3n. For the error condition: ResolvePath now only falls back on Arrow's scheme-mismatch error — any other PathFromUri failure (malformed URI, unsupported authority, …) is propagated — and it strips ?query/#fragment. oss/gs left for separate discussion, as you suggested.

…accept s3a/s3n

MisterRaindrop · 2026-06-08T10:29:22Z

+  }
+  // Scheme-less bucket/key, dropping any ?query / #fragment.
+  std::string bucket_key = file_location.substr(pos + 3);
+  return bucket_key.substr(0, bucket_key.find_first_of("?#"));


One subtle concern: s3:// and s3a:// currently use different parsing semantics. s3:// goes through Arrow's PathFromUri, while s3a/s3n use the fallback substring parser. This means percent-encoded keys may resolve differently.
example:
s3://bucket/a%20b -> Arrow PathFromUri -> bucket/a b
s3a://bucket/a%20b -> fallback -> bucket/a%20b

@MisterRaindrop Good catch — fixed. The fallback now percent-decodes the key via arrow::util::UriUnescape, so s3a/s3n resolve identically to native s3:// (a%20b → a b).

…rsing

MisterRaindrop · 2026-06-09T01:27:38Z

+
+  // Only fall back for Arrow's scheme-mismatch error; propagate anything else.
+  const auto& status = path.status();
+  if (status.message().find("expected a URI with one of the schemes") ==


This approach is problematic. It relies on Arrow's error messages, which will break if Arrow changes the message text. It does not simultaneously check status.code()

@MisterRaindrop Good point — dropped the error-text match. ResolvePath now validates the URI with arrow::util::Uri::Parse (a malformed URI fails there and propagates) and then percent-decodes the scheme-less bucket/key, so it no longer depends on Arrow's message text or status code.

…ror text

MisterRaindrop · 2026-06-09T11:47:43Z

Thanks for @plusplusjiajia contribution. There are a few minor issues with this PR.
I agree that the allowlist in DetectBuiltinFileIO covers the initial catalog-level FileIO selection from warehouse/io-impl.

My remaining concern is that later metadata/manifest/delete locations may reach the already-created FileIO directly, so
they are not necessarily guarded by DetectBuiltinFileIO. That said, I also agree that ResolvePath is not the right place to implement global scheme routing or rejection.

For the scope of this PR, this looks acceptable to me. I think we should follow up with a separate issue to discuss a higher-level resolving FileIO, similar to Java's ResolvingFileIO, for per-location scheme dispatch/enforcement.

feat(arrow): resolve S3-compatible schemes in ArrowFileSystemFileIO

293868e

wgtmac requested a review from Copilot June 8, 2026 07:00

Copilot started reviewing on behalf of wgtmac June 8, 2026 07:00 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

MisterRaindrop reviewed Jun 8, 2026

View reviewed changes

plusplusjiajia force-pushed the fix-arrow-s3-compatible-scheme branch 4 times, most recently from 0b65bf6 to c6886c7 Compare June 8, 2026 09:12

address review: scheme-mismatch-only fallback, strip query/fragment, …

5c7656b

…accept s3a/s3n

plusplusjiajia force-pushed the fix-arrow-s3-compatible-scheme branch from c6886c7 to 5c7656b Compare June 8, 2026 09:32

MisterRaindrop reviewed Jun 8, 2026

View reviewed changes

address review: percent-decode s3a/s3n fallback to match native s3 pa…

ef6c505

…rsing

MisterRaindrop reviewed Jun 9, 2026

View reviewed changes

plusplusjiajia force-pushed the fix-arrow-s3-compatible-scheme branch from bc30e66 to 0f32c67 Compare June 9, 2026 02:11

address review: parse URI via arrow::util::Uri instead of matching er…

277d8ba

…ror text

plusplusjiajia force-pushed the fix-arrow-s3-compatible-scheme branch from 0f32c67 to 277d8ba Compare June 9, 2026 02:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(arrow): resolve S3-compatible schemes in ArrowFileSystemFileIO#703

feat(arrow): resolve S3-compatible schemes in ArrowFileSystemFileIO#703
plusplusjiajia wants to merge 4 commits into
apache:mainfrom
plusplusjiajia:fix-arrow-s3-compatible-scheme

plusplusjiajia commented Jun 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

plusplusjiajia Jun 8, 2026 •

edited

Loading

Uh oh!

MisterRaindrop Jun 8, 2026 •

edited

Loading

Uh oh!

plusplusjiajia Jun 8, 2026 •

edited

Loading

Uh oh!

MisterRaindrop Jun 8, 2026

Uh oh!

plusplusjiajia Jun 8, 2026

Uh oh!

MisterRaindrop Jun 9, 2026 •

edited

Loading

Uh oh!

plusplusjiajia Jun 9, 2026 •

edited

Loading

Uh oh!

MisterRaindrop commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

plusplusjiajia commented Jun 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

plusplusjiajia Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MisterRaindrop commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

plusplusjiajia Jun 8, 2026 •

edited

Loading

MisterRaindrop Jun 8, 2026 •

edited

Loading

plusplusjiajia Jun 8, 2026 •

edited

Loading

MisterRaindrop Jun 9, 2026 •

edited

Loading

plusplusjiajia Jun 9, 2026 •

edited

Loading