Skip to content

feat(ipc): Supports compression level configuration#10133

Merged
Jefffrey merged 5 commits into
apache:mainfrom
wForget:ARROW-RS-10132
Jun 27, 2026
Merged

feat(ipc): Supports compression level configuration#10133
Jefffrey merged 5 commits into
apache:mainfrom
wForget:ARROW-RS-10132

Conversation

@wForget

@wForget wForget commented Jun 12, 2026 •
edited by Jefffrey
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Make compression level of ipc writer configurable

What changes are included in this PR?

  • Add a compression level option to IpcWriteOptions
  • Add try_with_compression_level to configure IPC batch compression levels
  • Pass the configured compression level through the IPC write path when constructing the compression codec

Are these changes tested?

I locally modified the test_write_file_with_zstd_compression test case to benchmark write time and output file size across different compression levels.

image

print logs:

ZSTD compression level -999: wrote 4001530 bytes in 1.658125ms
ZSTD compression level 1: wrote 3657530 bytes in 38.953042ms
ZSTD compression level 22: wrote 3638762 bytes in 869.19925ms

Are there any user-facing changes?

Yes. This PR adds a new batch_compression_level option to IpcWriteOptions. No public API changes

@github-actions github-actions Bot added the arrow Changes to the arrow crate label Jun 12, 2026
@wForget
wForget marked this pull request as ready for review June 12, 2026 07:32
@wForget

This comment was marked as outdated.

Sign in to view
@wForget wForget mentioned this pull request Jun 23, 2026
Open
Jefffrey
Jefffrey reviewed Jun 24, 2026
View reviewed changes
Comment thread arrow-ipc/src/compression.rs Outdated
Comment thread arrow-ipc/src/writer.rs Outdated
}

match (self.batch_compression_type, self.batch_compression_level) {
(Some(crate::CompressionType::ZSTD), Some(level)) if !(-999..=22).contains(&level) => {

@Jefffrey Jefffrey Jun 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use this for the range check instead?

@wForget wForget Jun 25, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, changed

wForget and others added 2 commits June 25, 2026 10:11
@wForget @Jefffrey
Apply suggestion from @Jefffrey
ef6e75d 
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
@wForget
address comment
cb9ca87
@wForget
wForget requested a review from Jefffrey June 26, 2026 03:03
@Jefffrey
Jefffrey merged commit 8de5640 into apache:main Jun 27, 2026
31 checks passed
@Jefffrey

Jefffrey commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

thanks @wForget

@Jefffrey Jefffrey mentioned this pull request Jun 27, 2026
Closed
@alamb alamb mentioned this pull request Jul 2, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jefffrey Jefffrey Jefffrey approved these changes

Assignees

No one assigned

Labels

arrow Changes to the arrow crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

arrow-ipc: Supports compression level configuration for arrow-ipc writer

2 participants

@wForget @Jefffrey