Commit 87383f3
Avoid the usage of intermediate ScalarValue to improve performance of extracting statistics from parquet files (apache#10711)
* Fix incorrect statistics read for unsigned integers columns in parquet
* Staging the change for faster stat
* Improve performance of extracting statistics from parquet files
* Revert "Improve performance of extracting statistics from parquet files"
This reverts commit 2faec57.
* Revert "Staging the change for faster stat"
This reverts commit 095ac39.
* Refine using the iterator idea
* Add the rest types
* Consolidate Decimal statistics extraction
* clippy
* Simplify
* Fix dictionary type
* Fix incorrect statistics read for timestamp columns in parquet
* Add exhaustive match
* Update latest datatypes
* fix bad comment
* Remove duplications using paste
* Fix comment
* Update Cargo.lock
* fix docs
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>1 parent 19845a3 commit 87383f3
4 files changed
Lines changed: 463 additions & 222 deletions
File tree
- datafusion-cli
- datafusion/core
- src/datasource/physical_plan/parquet
- tests/parquet
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
| 124 | + | |
124 | 125 | | |
125 | 126 | | |
126 | 127 | | |
| |||
0 commit comments