Supporting authorized S3 gateway redirects to presigned backing objects

[anonx3247]

Use case

An S3-compatible gateway authenticates and authorizes a logical object GET, then returns HTTP 307 to a short-lived presigned URL for the physical backing-S3 object. Range bytes then travel directly from backing storage to the client instead of being proxied through the authorization gateway.

This is intentional gateway behavior, not a region or endpoint misconfiguration. It resembles the Hugging Face S3-to-CDN flow fixed in apache/arrow-rs-object-store#790.

Current behavior

Lance currently pins object_store 0.13.2. Its SigV4 signer inserts an explicit Host header into the outgoing request. reqwest carries that Host across a cross-authority redirect, causing the presigned destination request to fail. object_store 0.14.1 fixes Host at the signing owner.

Lance cannot upgrade directly today because object_store_opendal 0.58 requires object_store ^0.13.1 and no compatible adapter release is available.

There is also a credential-boundary question: reqwest strips Authorization on a host change, but does not strip AWS session-token headers, S3 Express session tokens, SSE-C keys, or arbitrary headers.* gateway credentials.

Options

1. Wait until Lance can upgrade to object_store 0.14.1, and treat credential-safe redirect policy as an object_store concern.
2. Add a guarded Lance opt-in limited to static AWS credentials with no custom headers, explicitly rejecting S3 Express, temporary sessions, and SSE-C.
3. Declare cross-authority S3 redirects unsupported in Lance even after the object_store upgrade.

PR #8205 implemented option 2 and includes an end-to-end signed Range GET test with metrics/AIMD composition, but was closed pending this ownership discussion.

Which layer should own this behavior, and is this gateway pattern something Lance intends to support?

[anonx3247]

Additional dependency context: Apache OpenDAL already merged object_store 0.14.1 support in apache/opendal#7935 and published object_store_opendal 0.58.1, but that crate version was yanked because it broke existing 0.58 users by changing the object_store major compatibility range (apache/opendal#8008). So the signing-owner fix should become consumable once OpenDAL publishes a correctly versioned replacement; this strengthens option 1 if Lance does not want a local redirect policy.

[Xuanwo]

Let just Wait until Lance can upgrade to object_store 0.14.1

[anonx3247]

Hey sounds great, big fans of LanceDB btw, any idea what the ETA would be on lance supporting this upgrade? If there are any blockers we'd love to help!