Route Arrow - DType extension conversion through the registry by palaska · Pull Request #7592 · vortex-data/vortex

palaska · 2026-04-22T13:25:21Z

Arrow <-> DType extension conversion now consults the DTypeSession extension registry.
Any registered extension round-trips through Arrow by default, not just temporal types and Parquet Variant.
Extensions can register an Arrow canonical alias; FixedShapeTensor uses this to emit arrow.fixed_shape_tensor with JSON metadata, so pyarrow sees it as a first-class FixedShapeTensorArray.

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

codspeed-hq · 2026-04-22T15:58:26Z

Merging this PR will degrade performance by 10.6%

⚡ 24 improved benchmarks
❌ 1 regressed benchmark
✅ 1105 untouched benchmarks
⏩ 33 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`take_search[(0.005, 0.05)]`	167.6 µs	130.7 µs	+28.18%
⚡	Simulation	`take_search[(0.005, 0.1)]`	319.7 µs	246.5 µs	+29.72%
⚡	Simulation	`take_search[(0.005, 0.5)]`	1.5 ms	1.2 ms	+31.07%
⚡	Simulation	`take_search[(0.005, 1.0)]`	3.1 ms	2.3 ms	+31.28%
⚡	Simulation	`take_search[(0.01, 0.1)]`	340.6 µs	267.4 µs	+27.39%
⚡	Simulation	`take_search[(0.01, 0.05)]`	178.7 µs	141.8 µs	+25.97%
⚡	Simulation	`take_search[(0.01, 0.5)]`	1.6 ms	1.3 ms	+28.61%
⚡	Simulation	`take_search[(0.01, 1.0)]`	3.3 ms	2.5 ms	+28.79%
⚡	Simulation	`take_search[(0.1, 0.05)]`	248.4 µs	211.6 µs	+17.42%
⚡	Simulation	`take_search[(0.1, 0.1)]`	458.1 µs	384.8 µs	+19.06%
⚡	Simulation	`take_search[(0.1, 1.0)]`	4.3 ms	3.5 ms	+20.65%
⚡	Simulation	`take_search_chunked[(0.005, 0.1)]`	383.3 µs	321.1 µs	+19.37%
⚡	Simulation	`take_search_chunked[(0.005, 0.5)]`	1.9 ms	1.5 ms	+20.08%
⚡	Simulation	`take_search_chunked[(0.01, 0.05)]`	212.6 µs	181.4 µs	+17.23%
⚡	Simulation	`take_search[(0.1, 0.5)]`	2.2 ms	1.8 ms	+20.43%
⚡	Simulation	`take_search_chunked[(0.005, 1.0)]`	3.7 ms	3.1 ms	+20.18%
⚡	Simulation	`take_search_chunked[(0.005, 0.05)]`	199.9 µs	168.6 µs	+18.54%
⚡	Simulation	`take_search_chunked[(0.01, 0.5)]`	2 ms	1.7 ms	+18.64%
⚡	Simulation	`take_search_chunked[(0.01, 0.1)]`	407.9 µs	345.7 µs	+17.99%
⚡	Simulation	`take_search_chunked[(0.1, 0.05)]`	278.6 µs	247.3 µs	+12.66%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

_{Comparing bp/arrow-ext (fdaf5d7) with develop (140eec6)}

33 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-04-22T16:16:51Z

+/// Vortex-internal extension ids and Arrow canonical extension names. Canonical extensions
+/// serialize metadata as raw UTF-8 (typically JSON) rather than base64-wrapped bytes.
+const CANONICAL_ALIASES: &[(&str, &str)] =
+    &[("vortex.fixed_shape_tensor", "arrow.fixed_shape_tensor")];


Whoops I forgot to change vortex.fixed_shape_tensor to vortex.tensor.fixed_shape_tensor... Can you add that to this PR? I could do it myself but then these won't be synced. Maybe the str ID should just be a constant and both call sites use that?

AdamGS · 2026-04-22T16:27:51Z

ok I have thoughts but I'm going to head out soon and this required more thinking, I'll try and post a review first thing tomorrow @palaska

AdamGS · 2026-04-23T15:07:46Z

+const ARROW_EXT_NAME_VARIANT: &str = "arrow.parquet.variant";
+
+/// `(vortex_id, arrow_canonical_name)` pairs — single source of truth for bijection between
+/// Vortex-internal extension ids and Arrow canonical extension names. Canonical extensions
+/// serialize metadata as raw UTF-8 (typically JSON) rather than base64-wrapped bytes.
+const CANONICAL_ALIASES: &[(&str, &str)] =
+    &[("vortex.fixed_shape_tensor", "arrow.fixed_shape_tensor")];


Just putting what we talked about over slack here - I think we should avoid hard-coding extensions here, this whole part of the code shouldn't know about them and while we can have some best-practices/defaults or assumed semantics for some of these, it should be done externally and allow for customization by the user.

AdamGS · 2026-04-23T15:08:43Z

+/// resolve `ARROW:extension:name` metadata into [`DType::Extension`] values.
+///
+/// Unregistered or malformed extension metadata falls back to the storage dtype.
+pub trait FromArrowWithSession<T>: Sized {


Should this be a function with a default impl on TryFromArrowType?

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

joseph-isaacs · 2026-04-24T12:24:14Z

 #[derive(Debug)]
 pub struct DTypeSession {
    registry: ExtDTypeRegistry,
+    arrow_canonical: RwLock<ArrowCanonicalAliases>,


We cannot use a RwLock, it has very high contention. It rarely written to?

Claude recommended ArcSwap<HashMap>, lock-free reads and atomic write swaps

Yes, I think wrap it in a newtype just like #7588

joseph-isaacs · 2026-04-24T12:25:38Z

 //! Test extension types for exercising the [`ExtVTable`] contract.

-mod divisible_int;
+pub(crate) mod divisible_int;


for the newly added extension_roundtrip tests, i needed a test extension type with metadata. What is your comment here? 😄

It was do we need to do this

joseph-isaacs · 2026-04-24T12:27:55Z

Why remove this in this PR

arrow requires metadata to be UTF-8 json so i swapped, do we want to keep the proto around?

@connortsui20 is this okay?

But in general JSON is less space efficient than proto

I think it's fine so that we are compat with arrow

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska · 2026-04-27T11:58:22Z

+    match field.extension_type_metadata() {
+        None | Some("") => Ok(Cow::Borrowed(&[])),
+        Some(s) if is_canonical => Ok(Cow::Borrowed(s.as_bytes())),
+        Some(s) => BASE64_STANDARD


@joseph-isaacs should I get rid of this and assert valid utf 8 on extension metadata serde? Then this codepath will be simpler (String::from_utf8) and we can make serialize_metadata return VortexResult<String> instead of bytes.

Do we store BASE_64 json in the vortex file for this type? Surely that is a waste of space

this is only the arrow serde path. Metadata for the fixed shape tensor is utf8 encoded json bytes, no base64. I guess its still larger than proto but it's metadata at the end of the day, your call :)

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

…ndary Signed-off-by: Baris Palaska <barispalaska@gmail.com>

…arrow-ext

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

…arrow-ext

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

…arrow-ext

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska changed the title ~~Route Arrow ↔ DType extension conversion through the registry~~ Route Arrow - DType extension conversion through the registry Apr 22, 2026

palaska added 2 commits April 22, 2026 16:53

arrow dtype extension conversion

8423533

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

use arrow canonical extension name

47efbdb

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska force-pushed the bp/arrow-ext branch from 7d33d8b to 47efbdb Compare April 22, 2026 15:54

palaska added the changelog/feature A new feature label Apr 22, 2026

palaska marked this pull request as ready for review April 22, 2026 15:59

palaska requested a review from connortsui20 April 22, 2026 15:59

connortsui20 requested a review from AdamGS April 22, 2026 16:09

connortsui20 reviewed Apr 22, 2026

View reviewed changes

AdamGS reviewed Apr 23, 2026

View reviewed changes

externalize canonical aliases

ad80b53

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska force-pushed the bp/arrow-ext branch from bdcbb5d to ad80b53 Compare April 24, 2026 11:24

reuse ids

c4d1079

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska requested review from AdamGS and connortsui20 April 24, 2026 12:21

joseph-isaacs reviewed Apr 24, 2026

View reviewed changes

Comment thread vortex-array/src/dtype/session.rs Outdated

joseph-isaacs reviewed Apr 24, 2026

View reviewed changes

Comment thread vortex-tensor/src/types/fixed_shape/mod.rs Outdated

joseph-isaacs reviewed Apr 24, 2026

View reviewed changes

Comment thread vortex-tensor/src/lib.rs Outdated

joseph-isaacs reviewed Apr 24, 2026

View reviewed changes

Comment thread vortex-tensor/src/types/vector/vtable.rs Outdated

palaska added 5 commits April 27, 2026 09:33

cached id

62d70e2

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

arcswap

c3c455d

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

merge

426b8dd

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

clippy

a91c861

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'develop' into bp/arrow-ext

aa042f3

refactor tests, dont copy metadata bytes

343139a

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska requested a review from joseph-isaacs April 27, 2026 11:26

palaska added 2 commits April 27, 2026 12:41

rm unnecessary stuff

fd6654b

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

better name

80f52f1

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

palaska commented Apr 27, 2026

View reviewed changes

palaska added 18 commits April 27, 2026 15:55

newtype, static ID

784d283

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'develop' into bp/arrow-ext

b291a5c

keep on-disk metadata as proto, convert to json only at the arrow bou…

3b67a4d

…ndary Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'bp/arrow-ext' of github.com:vortex-data/vortex into bp/…

dc8bd4a

…arrow-ext

revert comments

1ddb32b

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

public api, fmt

76ef82e

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'develop' into bp/arrow-ext

498830e

refactor registry

2978add

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'develop' into bp/arrow-ext

ff7e1ac

nit

b61239c

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'bp/arrow-ext' of github.com:vortex-data/vortex into bp/…

ce9b58b

…arrow-ext

unwrap extension to storage in execute_arrow

2b2a9a3

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

clippy

2155f20

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'develop' into bp/arrow-ext

69838a9

recover extension identity in FromArrowArray

990a4b8

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Merge branch 'bp/arrow-ext' of github.com:vortex-data/vortex into bp/…

c6333ef

…arrow-ext

Merge branch 'develop' into bp/arrow-ext

0923b1b

clippy

fdaf5d7

Signed-off-by: Baris Palaska <barispalaska@gmail.com>

Conversation

palaska commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 10.6%

Performance Changes

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AdamGS commented Apr 22, 2026

Uh oh!

AdamGS Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

palaska Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joseph-isaacs Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

palaska commented Apr 22, 2026 •

edited

Loading

codspeed-hq Bot commented Apr 22, 2026 •

edited

Loading

AdamGS Apr 23, 2026 •

edited

Loading

palaska Apr 27, 2026 •

edited

Loading

joseph-isaacs Apr 27, 2026 •

edited

Loading