Skip to content

Commit 87383f3

Browse files
xinlifoobaralamb
authored andcommitted
Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files (apache#10711)
* Fix incorrect statistics read for unsigned integers columns in parquet * Staging the change for faster stat * Improve performance of extracting statistics from parquet files * Revert "Improve performance of extracting statistics from parquet files" This reverts commit 2faec57. * Revert "Staging the change for faster stat" This reverts commit 095ac39. * Refine using the iterator idea * Add the rest types * Consolidate Decimal statistics extraction * clippy * Simplify * Fix dictionary type * Fix incorrect statistics read for timestamp columns in parquet * Add exhaustive match * Update latest datatypes * fix bad comment * Remove duplications using paste * Fix comment * Update Cargo.lock * fix docs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
1 parent 19845a3 commit 87383f3

4 files changed

Lines changed: 463 additions & 222 deletions

File tree

datafusion-cli/Cargo.lock

Lines changed: 22 additions & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

datafusion/core/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ num_cpus = { workspace = true }
121121
object_store = { workspace = true }
122122
parking_lot = { workspace = true }
123123
parquet = { workspace = true, optional = true, default-features = true }
124+
paste = "1.0.15"
124125
pin-project-lite = "^0.2.7"
125126
rand = { workspace = true }
126127
sqlparser = { workspace = true }

0 commit comments

Comments
 (0)