Skip to content

Commit c1fd23f

Browse files
andygroveandygrove
authored
blog: Apache DataFusion Comet 0.17.0 release post (#198)
1 parent 5a8e6af commit c1fd23f

1 file changed

Lines changed: 245 additions & 0 deletions

File tree

‎content/blog/2026-06-20-datafusion-comet-0.17.0.md‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
---
2+
layout: post
3+
title: Apache DataFusion Comet 0.17.0 Release
4+
date: 2026-06-20
5+
author: pmc
6+
categories: [subprojects]
7+
---
8+
9+
<!--
10+
{% comment %}
11+
Licensed to the Apache Software Foundation (ASF) under one or more
12+
contributor license agreements. See the NOTICE file distributed with
13+
this work for additional information regarding copyright ownership.
14+
The ASF licenses this file to you under the Apache License, Version 2.0
15+
(the "License"); you may not use this file except in compliance with
16+
the License. You may obtain a copy of the License at
17+
18+
http://www.apache.org/licenses/LICENSE-2.0
19+
20+
Unless required by applicable law or agreed to in writing, software
21+
distributed under the License is distributed on an "AS IS" BASIS,
22+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
23+
See the License for the specific language governing permissions and
24+
limitations under the License.
25+
{% endcomment %}
26+
-->
27+
28+
[TOC]
29+
30+
The Apache DataFusion PMC is pleased to announce version 0.17.0 of the [Comet](https://datafusion.apache.org/comet/) subproject.
31+
32+
This release covers approximately five weeks of development work and is the result of merging 192 PRs from 19
33+
contributors. See the [change log] for more information.
34+
35+
[change log]: https://github.com/apache/datafusion-comet/blob/main/dev/changelog/0.17.0.md
36+
37+
## Fewer Fallbacks to Spark
38+
39+
The headline feature of 0.17.0 is a new mechanism that keeps more of your query running inside Comet instead
40+
of falling back to Spark: the **JVM codegen dispatcher**.
41+
42+
Comet has always fallen back to Spark row-based execution whenever an expression had no native Rust implementation, or where the
43+
Rust implementation could diverge from Spark on edge cases. A fallback is correct, but a columnar-to-row
44+
conversion is needed to feed the data into Spark's row-based operators, which adds overhead when processing billions of rows of data.
45+
46+
The codegen dispatcher avoids the fallback to row-based processing by running Spark's own
47+
generated code (`doGenCode`) inside the Comet pipeline, operating directly on Arrow batches. The result is a
48+
JVM-implemented Arrow-native expression: the data stays in Arrow format, and because the expression is
49+
evaluated by Spark's own code, the result is guaranteed to match Spark exactly across every supported Spark
50+
version. When the dispatcher is disabled, Comet falls back as before.
51+
52+
A dispatched expression is no faster than it would be in Spark, since it runs the same generated code. The
53+
benefit is that a single unsupported expression no longer forces an entire
54+
query stage back to row-based execution. The surrounding operators stay Arrow-native, and the
55+
stage avoids the columnar-to-row conversion and row-based Spark execution that a fallback would otherwise
56+
impose.
57+
58+
This release puts the dispatcher to work across a wide surface:
59+
60+
- **100% Spark-compatible regular expressions.** Regex expressions now dispatch to Spark's own
61+
implementation, eliminating the long-standing compatibility gaps of a separate native regex engine.
62+
- **100% Spark-compatible JSON functions.** JSON expression handling follows the same approach, matching
63+
Spark's behavior precisely.
64+
- **More scalar and structured-text functions**, including a batch of math and string functions, AES
65+
encryption and decryption (`aes_encrypt` / `aes_decrypt` / `try_aes_decrypt`), `Upper` / `Lower` /
66+
`InitCap`, `GetTimestamp`, `mask`, `try_to_number`, and an expanded set of date, time, and
67+
timezone expressions.
68+
- **Collection and higher-order expressions**, including `array_intersect`, `array_except`, `array_join`,
69+
`create_map`, and lambda-based higher-order functions such as `filter`.
70+
71+
0.17.0 also changes how Comet treats expressions whose native Rust path is known to diverge
72+
from Spark (marked `Incompatible`). Previously such an expression forced the entire projection back to Spark
73+
unless the user opted into the divergent native behavior with the per-expression
74+
`spark.comet.expression.<name>.allowIncompatible` flag. By default, that flag is unset, and the expression is
75+
now routed through the codegen dispatcher and evaluated correctly inside Comet rather than triggering a
76+
fallback. Setting it to `true` becomes a performance knob for users who accept the faster native path's
77+
divergence. Expressions such as `from_unixtime`, and the `TimestampNTZ` branches of `hour`, `minute`, and
78+
`second`, now stay in the pipeline by default.
79+
80+
## User-Defined Functions in Java and Scala
81+
82+
Building on the codegen dispatcher, 0.17.0 adds support for arbitrary user-defined functions written in Java
83+
and Scala, enabled by default. Eligible Spark `ScalaUDF` expressions are routed through the dispatcher and
84+
executed inside the Comet pipeline, so a project around a UDF no longer forces a fallback and a
85+
columnar-to-row conversion.
86+
87+
The path has broad type coverage — scalars, arbitrarily nested complex types, and higher-order functions — and
88+
is backed by end-to-end, fuzz, and Iceberg test coverage. It can be disabled with
89+
`spark.comet.exec.scalaUDF.codegen.enabled=false`.
90+
91+
## Arrow-Native, End to End
92+
93+
The codegen dispatcher works because of a property that runs through all of Comet: queries stay **Arrow-native
94+
end to end**. Operators, expressions, shuffle, and broadcast all remain in Apache Arrow columnar format,
95+
avoiding the per-row overhead that Spark's row-based engine incurs from materializing and transitioning data
96+
one row at a time.
97+
98+
Within that Arrow-native pipeline, the work of an operator or expression runs in one of two ways:
99+
100+
- **Rust-implemented**: native Rust code, executed through Apache DataFusion. This is what most people picture
101+
when they think of Comet.
102+
- **JVM-implemented**: Scala or Java code that operates directly on Arrow batches, including the
103+
codegen-dispatched expressions and UDFs described above.
104+
105+
Because the data never leaves Arrow columnar format, there is no per-row materialization cost at the boundary
106+
between a Rust-implemented and a JVM-implemented step. A dispatched expression sits in the pipeline alongside
107+
Rust-implemented operators with no columnar-to-row transition between them. Expanding the JVM-implemented path
108+
in 0.17.0 lets Comet keep more work Arrow-native instead of handing it back to Spark.
109+
110+
## Expanded Expression Coverage
111+
112+
Partly through the codegen dispatcher and partly through new Rust implementations, Comet's expression coverage
113+
grew substantially in this release. More than 120 Spark expressions have gained support since 0.16.0, across
114+
most function families:
115+
116+
- **Date and time** (~25): `convert_timezone`, `make_date`, `months_between`, `next_day`,
117+
`from_utc_timestamp` / `to_utc_timestamp`, `date_from_unix_date`, the `timestamp_*` and `unix_*` second /
118+
milli / micro conversions, and the current date/time/timezone functions.
119+
- **Math** (~24): `acosh`, `asinh`, `atanh`, `cbrt`, `csc`, `sec`, `hypot`, `log1p`, `bin`, `conv`,
120+
`factorial`, `pmod`, `width_bucket`, `rint`, and more.
121+
- **String** (~16): `elt`, `find_in_set`, `format_number`, `format_string`, `levenshtein`, `locate`,
122+
`overlay`, `soundex`, `split`, `substring_index`, `unbase64`, `to_char`, and `to_number`.
123+
- **XPath** (9): the full `xpath`, `xpath_boolean`, `xpath_double`, `xpath_int`, `xpath_long`, `xpath_string`
124+
family.
125+
- **JSON, CSV, and XML** (9): `from_csv`, `to_csv`, `schema_of_csv`, `schema_of_json`, `json_object_keys`,
126+
`json_array_length`, `from_xml`, `to_xml`, and `schema_of_xml`.
127+
- **Array and map** (11): `array_position`, `array_size`, `arrays_zip`, `slice`, `sort_array`, `sequence`,
128+
`map_concat`, `map_contains_key`, and `map_from_entries`.
129+
- **Aggregate and window** (7): `any_value`, `count_if`, the `regr_*` regression aggregates, plus `lag` and
130+
`lead`.
131+
- **Conditional and null handling** (7): `greatest`, `least`, `nullif`, `ifnull` / `nvl`, `nvl2`, and
132+
`equal_null`.
133+
134+
For the full list of supported expressions in this release, see the
135+
[Spark Expression Support](https://datafusion.apache.org/comet/user-guide/0.17/expressions.html) reference.
136+
137+
Operator coverage grew too: 0.17.0 adds a native **broadcast nested loop join**, so queries with non-equi or
138+
cross join conditions can now stay in the Comet pipeline rather than falling back to Spark.
139+
140+
## Performance
141+
142+
### Removing an FFI Round Trip from Native Shuffle
143+
144+
The most significant performance change in 0.17.0 targets the shuffle write path. When a native subtree feeds
145+
a Comet shuffle, Comet previously ran two separate native iterators per partition: one for the upstream
146+
subtree, and a second rooted at a synthetic `Scan("ShuffleWriterInput") -> ShuffleWriter` that consumed the
147+
batches back. The JVM never actually read this data, so the native-to-JVM-and-back hop was pure overhead.
148+
149+
Although the Arrow C Data Interface is zero-copy, this particular round trip was not. The synthetic scan left
150+
its batches marked as not FFI-safe, so on import every batch was deep-copied into freshly allocated buffers.
151+
0.17.0 collapses the two iterators into a single native plan rooted at the shuffle writer, with the upstream
152+
subtree as a direct child. That removes the Arrow FFI export and import, the per-batch deep copy, and one
153+
`createPlan` / `releasePlan` pair per partition, while preserving all of the existing per-partition setup
154+
(broadcast alignment, subqueries, encryption) and input metrics reporting. The gain comes from removing the
155+
round trip, not from copying its data more efficiently.
156+
157+
### A Single Arrow Stream on the Input Path
158+
159+
0.17.0 also reworks the other side of the JVM/native boundary — the path that feeds JVM-sourced data _into_
160+
native execution. Previously each batch crossed via a bespoke `CometBatchIterator`, with a `hasNext` / `next`
161+
JNI pair per batch and every column imported through its own Arrow FFI array and schema. This release replaces
162+
that path with the **Arrow C Stream Interface**: the JVM exports each per-partition iterator once, and native
163+
imports the schema once and pulls each batch through a single C callback, taking ownership by reference count.
164+
This removes the per-batch, per-column FFI export and JNI round trips for all JVM-sourced inputs, lets the
165+
old `CometBatchIterator` and a now-unnecessary deep copy be deleted, and is the input-side counterpart to the
166+
shuffle-write change above. TPC-DS at 1TB is about 9% faster in 0.17.0 than in 0.16.0, and these two FFI
167+
changes are the largest contributor to that gain.
168+
169+
### Lower Per-Batch Overhead in Arrow Vectors
170+
171+
Two changes reduce repeated work when reading Arrow columns across the JNI boundary. Comet now caches the
172+
validity buffer address on `CometDecodedVector` and the offset buffer address for variable-width vectors on
173+
`CometPlainVector`, so these addresses are resolved once per vector rather than on every access.
174+
175+
### Faster Statistical Aggregates
176+
177+
The variance, standard deviation, covariance, and correlation aggregates now use DataFusion's
178+
`GroupsAccumulator` interface, which is substantially more efficient for grouped aggregation than the
179+
row-accumulator path they used previously.
180+
181+
Additional smaller improvements include bulk-NULL handling in `split` and `substring` (skipping a per-row
182+
allocation), and a new `interleave_time` shuffle metric with tuned output buffer sizing to make shuffle cost
183+
easier to attribute.
184+
185+
## Preparing for the 1.0.0 Release
186+
187+
Much of the correctness work in this release is part of an ongoing push toward a **1.0.0 release**.
188+
Planning for 1.0.0 is being tracked in [issue #4082], where the community is discussing what the milestone
189+
should mean. The criteria under consideration go well beyond correctness and include:
190+
191+
- Demonstrated cost savings for TPC-H and TPC-DS at SF1000 (1TB)
192+
- Thorough documentation of compatibility status
193+
- A review of all configuration options, renaming some for consistency
194+
- Consistent logging
195+
- A documented policy for how long each Spark version will be supported
196+
- A documented process for preventing major performance regressions
197+
- A documented semantic-versioning policy and what it means for Comet going forward
198+
199+
[issue #4082]: https://github.com/apache/datafusion-comet/issues/4082
200+
201+
Correctness is central to that list: reaching 1.0.0 means being able to state precisely which Spark
202+
expressions Comet accelerates, and how faithfully it matches Spark on each one. Two efforts in 0.17.0 move
203+
toward that goal.
204+
205+
First, the codegen dispatcher guarantees an exact match for every dispatched expression, since Spark's own
206+
generated code does the evaluation. That raises our confidence that Comet matches Spark across the supported
207+
versions, and it makes the Spark Expression Support reference something the project can stand behind as it
208+
approaches 1.0.0.
209+
210+
Second, this release included a systematic audit of Comet's existing expression implementations against Spark,
211+
detailed below.
212+
213+
### Expression Audit
214+
215+
The audit compared Comet's expression implementations against Apache Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1,
216+
covering the hash, JSON, collection, map, predicate, bitwise, conditional, array, struct, math, and cast
217+
expression families, along with cast behavior. Each audit compared Comet's behavior to Spark across all four
218+
versions and expanded test coverage where gaps were found.
219+
220+
This edge-case-by-edge-case comparison surfaced most of the behavior differences fixed in this
221+
release. By working through each expression's corner cases and writing targeted Comet SQL tests for them, the
222+
audit caught divergences that broader testing does not reliably exercise. Comet also runs the full Apache
223+
Spark SQL test suite against each supported Spark version as part of CI, which continues to catch some
224+
cross-version differences, but the audit's focused approach found considerably more.
225+
226+
## Compatibility
227+
228+
Supported platforms include:
229+
230+
- **Spark 3.4.3** with Java 11/17 and Scala 2.12/2.13
231+
- **Spark 3.5.8** with Java 11/17 and Scala 2.12/2.13
232+
- **Spark 4.0.2** with Java 17 and Scala 2.13
233+
- **Spark 4.1.2** with Java 17 and Scala 2.13
234+
235+
See the [Spark Version Compatibility] page for known limitations specific to each version.
236+
237+
[Spark Version Compatibility]: https://datafusion.apache.org/comet/user-guide/latest/compatibility/spark-versions.html
238+
239+
This release builds on **DataFusion 53.1** and **Arrow 58.3**.
240+
241+
## Get Started with Comet 0.17.0
242+
243+
Ready to try it out? Follow the [Comet 0.17.0 Installation Guide](https://datafusion.apache.org/comet/user-guide/0.17/installation.html)
244+
to get up and running, then point Comet at your existing Spark workloads, including Scala and Java UDFs and
245+
Spark 4 with ANSI mode enabled, and see the speedup for yourself.

0 commit comments

Comments
 (0)