Skip to content

Commit 2a77e10

Browse files
authored
feat(ipc): add with_skip_validation to StreamDecoder (#9749)
# Which issue does this PR close? N/A - this is a small API parity fix. # Rationale for this change `StreamReader` and `FileDecoder` both expose `with_skip_validation` but `StreamDecoder` does not, despite having the same internal `skip_validation` field. Motivation: DataFusion's spill infrastructure is being refactored to use `StreamDecoder` for reading spill files via a pluggable backend trait ([apache/datafusion#21215](apache/datafusion#21215)). Since DataFusion controls what it writes it can trust its own IPC output, and needs this method to maintain the same performance characteristic as the current StreamReader-based implementation # What changes are included in this PR? Adds `with_skip_validation` to `StreamDecoder` mirroring the existing implementation on `FileDecoder` # Are these changes tested? The existing StreamDecoder tests cover the decoding path. No new tests added as this method sets an internal flag already tested via `FileDecoder` and `StreamReader`. # Are there any user-facing changes? Yes, `with_skip_validation` is now available on `StreamDecoder`. It is marked unsafe with the same safety requirements as `FileDecoder::with_skip_validation`.
1 parent b93240a commit 2a77e10

1 file changed

Lines changed: 15 additions & 2 deletions

File tree

arrow-ipc/src/reader/stream.rs

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,8 @@ pub struct StreamDecoder {
4545
require_alignment: bool,
4646
/// Should validation be skipped when reading data? Defaults to false.
4747
///
48-
/// See [`FileDecoder::with_skip_validation`] for details.
48+
/// See [`StreamDecoder::with_skip_validation`] for details.
4949
///
50-
/// [`FileDecoder::with_skip_validation`]: crate::reader::FileDecoder::with_skip_validation
5150
skip_validation: UnsafeFlag,
5251
}
5352

@@ -114,6 +113,20 @@ impl StreamDecoder {
114113
self.schema.as_ref().map(|schema| schema.clone())
115114
}
116115

116+
/// Specifies if validation should be skipped when reading data (defaults to `false`)
117+
///
118+
/// # Safety
119+
///
120+
/// This flag must only be set to `true` when you trust the input data and are
121+
/// sure the data you are reading is valid Arrow IPC stream data, otherwise
122+
/// undefined behavior may result.
123+
///
124+
/// For example, DataFusion uses this when reading spill files it wrote itself.
125+
pub unsafe fn with_skip_validation(mut self, skip_validation: bool) -> Self {
126+
unsafe { self.skip_validation.set(skip_validation) };
127+
self
128+
}
129+
117130
/// Try to read the next [`RecordBatch`] from the provided [`Buffer`]
118131
///
119132
/// [`Buffer::advance`] will be called on `buffer` for any consumed bytes.

0 commit comments

Comments
 (0)