`get_slice_memory_size` underestimates memory for `StringViewArray`

[ding-young]

Describe the bug

It is found (in apache/datafusion#17029) that get_sliced_memory_size underestimated the memory size for StringViewArray. The diff was quite large i.e. when correct batch size was 1392730 bytes, it returned only 163840 bytes.

This is because payload buffers are not taken account of although these payload buffers are simply cloned() when we slice() StringViewArray

See

arrow-rs/arrow-data/src/data.rs

Lines 464 to 476 in a620957

	pub fn get_slice_memory_size(&self) -> Result<usize, ArrowError> {
	let mut result: usize = 0;
	let layout = layout(&self.data_type);
	for spec in layout.buffers.iter() {
	match spec {
	BufferSpec::FixedWidth { byte_width, .. } => {
	let buffer_size = self.len.checked_mul(*byte_width).ok_or_else(|| {
	ArrowError::ComputeError(
	"Integer overflow computing buffer size".to_string(),
	)
	})?;
	result += buffer_size;

arrow-rs/arrow-data/src/data.rs

Lines 1743 to 1752 in a620957

	pub fn new_view() -> Self {
	Self {
	buffers: vec![BufferSpec::FixedWidth {
	byte_width: mem::size_of::<u128>(),
	alignment: mem::align_of::<u128>(),
	}],
	can_contain_null_mask: true,
	variadic: true,
	}
	}

To Reproduce

Expected behavior

get_sliced_memory_size should include the capacity for payload buffers.
If this fix isn’t ideal, we should at least document which buffers are being counted in the calculation.

Additional context

apache/datafusion#17315
here's related pr for temporal fix in datafusion

[ding-young]

take

[Jefffrey]

some pointers

arrow-rs/arrow-data/src/data.rs

Lines 514 to 589 in 12091af

	/// Returns the total number of the bytes of memory occupied by
	/// the buffers by this slice of [`ArrayData`] (See also diagram on [`ArrayData`]).
	///
	/// This is approximately the number of bytes if a new
	/// [`ArrayData`] was formed by creating new [`Buffer`]s with
	/// exactly the data needed.
	///
	/// For example, a [`DataType::Int64`] with `100` elements,
	/// [`Self::get_slice_memory_size`] would return `100 * 8 = 800`. If
	/// the [`ArrayData`] was then [`Self::slice`]ed to refer to its
	/// first `20` elements, then [`Self::get_slice_memory_size`] on the
	/// sliced [`ArrayData`] would return `20 * 8 = 160`.
	pub fn get_slice_memory_size(&self) -> Result<usize, ArrowError> {
	let mut result: usize = 0;
	let layout = layout(&self.data_type);
	for spec in layout.buffers.iter() {
	match spec {
	BufferSpec::FixedWidth { byte_width, .. } => {
	// Offset buffers contain len+1 elements: one boundary per element
	// plus a final boundary marking the end of the last element.
	let len = match self.data_type {
	DataType::Utf8
	| DataType::LargeUtf8
	| DataType::Binary
	| DataType::LargeBinary
	| DataType::List(_)
	| DataType::LargeList(_)
	| DataType::Map(_, _) => self.len + 1,
	_ => self.len,
	};
	let buffer_size = len.checked_mul(*byte_width).ok_or_else(|| {
	ArrowError::ComputeError(
	"Integer overflow computing buffer size".to_string(),
	)
	})?;
	result += buffer_size;
	}
	BufferSpec::VariableWidth => {
	let buffer_len = match self.data_type {
	DataType::Utf8 | DataType::Binary => {
	let offsets = self.typed_offsets::<i32>()?;
	(offsets[self.len] - offsets[0]) as usize
	}
	DataType::LargeUtf8 | DataType::LargeBinary => {
	let offsets = self.typed_offsets::<i64>()?;
	(offsets[self.len] - offsets[0]) as usize
	}
	_ => {
	return Err(ArrowError::NotYetImplemented(format!(
	"Invalid data type for VariableWidth buffer. Expected Utf8, LargeUtf8, Binary or LargeBinary. Got {}",
	self.data_type
	)));
	}
	};
	result += buffer_len;
	}
	BufferSpec::BitMap => {
	let buffer_size = bit_util::ceil(self.len, 8);
	result += buffer_size;
	}
	BufferSpec::AlwaysNull => {
	// Nothing to do
	}
	}
	}
	if self.nulls().is_some() {
	result += bit_util::ceil(self.len, 8);
	}
	for child in &self.child_data {
	result += child.get_slice_memory_size()?;
	}
	Ok(result)
	}

could likely add to the bottom, checking if we are a utf8/binary view type, and just take the sizes of the values buffers (could be multiple) unchanged since if we slice them we aren't gonna slice the values buffers