feat: add builder execution path with AppendChild step#7677
feat: add builder execution path with AppendChild step#7677joseph-isaacs merged 12 commits intodevelopfrom
Conversation
Merging this PR will degrade performance by 25.28%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | dict[10M_u64_values_u32_codes] |
318.7 µs | 279 µs | +14.21% |
| ❌ | WallTime | runend[10M_i32_runlen_10] |
159.5 µs | 195.1 µs | -18.24% |
| ❌ | WallTime | 10M_10%[1000000] |
215.9 µs | 249.3 µs | -13.4% |
| ❌ | WallTime | 10M_90%[10000000] |
195.2 µs | 226.3 µs | -13.77% |
| ❌ | WallTime | 10M_50%[5000000] |
278 µs | 312.1 µs | -10.91% |
| ❌ | Simulation | chunked_opt_bool_into_canonical[(10, 1000)] |
999 µs | 1,337.1 µs | -25.28% |
| ❌ | Simulation | chunked_opt_bool_into_canonical[(100, 100)] |
220.1 µs | 255.9 µs | -13.99% |
| ❌ | Simulation | chunked_varbinview_into_canonical[(10, 1000)] |
1.7 ms | 2 ms | -16.44% |
| ❌ | Simulation | chunked_varbinview_opt_into_canonical[(10, 1000)] |
2.5 ms | 2.8 ms | -11.02% |
| ⚡ | Simulation | decompress[alp_for_bp_f64] |
2.9 ms | 1.8 ms | +58.41% |
| ⚡ | Simulation | alp_rd_decompress_f64 |
2.4 ms | 1.1 ms | ×2.2 |
| ⚡ | Simulation | decompress_rd[f32, (100000, 0.1)] |
1,374.4 µs | 582.4 µs | ×2.4 |
| ⚡ | Simulation | decompress_rd[f32, (10000, 0.1)] |
165.6 µs | 81.7 µs | ×2 |
| ⚡ | Simulation | decompress_rd[f64, (100000, 0.01)] |
2.3 ms | 1 ms | ×2.3 |
| ⚡ | Simulation | decompress_rd[f64, (10000, 0.1)] |
258.5 µs | 121.7 µs | ×2.1 |
| ⚡ | Simulation | decompress_rd[f64, (10000, 0.0)] |
258 µs | 121.8 µs | ×2.1 |
| ⚡ | Simulation | decompress_rd[f32, (100000, 0.01)] |
1,374.9 µs | 582.3 µs | ×2.4 |
| ⚡ | Simulation | decompress_rd[f32, (10000, 0.0)] |
165.9 µs | 85.5 µs | +93.95% |
| ⚡ | Simulation | decompress_rd[f64, (100000, 0.1)] |
2.3 ms | 1 ms | ×2.3 |
| ⚡ | Simulation | decompress_rd[f64, (100000, 0.0)] |
2,294.5 µs | 978.5 µs | ×2.3 |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Comparing ji/append-to-builder-iter (cbafa66) with develop (70eee88)
ddfb8fb to
2bdb707
Compare
2bdb707 to
b70fa7e
Compare
Benchmarks: CompressionVortex (geomean): 1.006x ➖ unknown / unknown (1.007x ➖, 6↑ 8↓)
|
Benchmarks: Random AccessVortex (geomean): 0.977x ➖ unknown / unknown (0.991x ➖, 2↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.022x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.020x ➖, 0↑ 0↓)
datafusion / parquet (1.034x ➖, 0↑ 0↓)
datafusion / arrow (1.070x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed (1.019x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.024x ➖, 0↑ 0↓)
duckdb / parquet (1.019x ➖, 0↑ 0↓)
duckdb / duckdb (1.017x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.027x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.017x ➖, 0↑ 0↓)
datafusion / parquet (1.037x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.993x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.014x ➖, 1↑ 1↓)
duckdb / parquet (1.042x ➖, 0↑ 1↓)
Full attributed analysis
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) duckdb / vortex-file-compressed (0.932x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.986x ➖, 0↑ 0↓)
duckdb / parquet (0.989x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.976x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.985x ➖, 0↑ 0↓)
datafusion / parquet (1.002x ➖, 0↑ 2↓)
datafusion / arrow (0.972x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed (0.993x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.997x ➖, 0↑ 0↓)
duckdb / parquet (0.987x ➖, 1↑ 0↓)
duckdb / duckdb (0.992x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: PolarSignals ProfilingVortex (geomean): 0.887x ✅ datafusion / vortex-file-compressed (0.887x ✅, 6↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (1.043x ➖, 2↑ 5↓)
datafusion / vortex-compact (1.045x ➖, 0↑ 7↓)
datafusion / parquet (1.049x ➖, 0↑ 12↓)
duckdb / vortex-file-compressed (1.041x ➖, 0↑ 9↓)
duckdb / vortex-compact (1.034x ➖, 0↑ 4↓)
duckdb / parquet (1.025x ➖, 0↑ 4↓)
duckdb / duckdb (1.048x ➖, 1↑ 9↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (0.897x ➖, 1↑ 1↓)
datafusion / vortex-compact (0.952x ➖, 1↑ 2↓)
datafusion / parquet (0.901x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.981x ➖, 0↑ 0↓)
duckdb / parquet (1.033x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) datafusion / vortex-file-compressed (1.024x ➖, 0↑ 2↓)
datafusion / vortex-compact (0.953x ➖, 0↑ 0↓)
datafusion / parquet (0.901x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (1.029x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.972x ➖, 0↑ 0↓)
duckdb / parquet (0.976x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: FineWeb S3Verdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.959x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.975x ➖, 0↑ 0↓)
datafusion / parquet (0.850x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.040x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.948x ➖, 1↑ 0↓)
duckdb / parquet (0.938x ➖, 0↑ 0↓)
Full attributed analysis
|
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) datafusion / vortex-file-compressed (0.976x ➖, 1↑ 1↓)
datafusion / parquet (0.969x ➖, 2↑ 0↓)
duckdb / vortex-file-compressed (0.978x ➖, 4↑ 2↓)
duckdb / parquet (0.995x ➖, 1↑ 1↓)
duckdb / duckdb (0.990x ➖, 1↑ 1↓)
Full attributed analysis
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (2)
Powered by Polar Signals Cloud |
File Sizes: PolarSignals ProfilingNo file size changes detected. |
File Sizes: FineWeb NVMeNo file size changes detected. |
File Sizes: TPC-H SF=1 on NVMEFile Size Changes (195 files changed, -98.4% overall, 0↑ 195↓)
Totals:
|
File Sizes: TPC-DS SF=1 on NVMENo file size changes detected. |
File Sizes: TPC-H SF=10 on NVMENo file size changes detected. |
File Sizes: Statistical and Population GeneticsNo file size changes detected. |
File Sizes: Clickbench on NVMEFile Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
robert3005
left a comment
There was a problem hiding this comment.
I think this mostly looks fine, there's a bit of additional refactoring here that is unrelated to the title of the pr which doesn't seem intentional
…er-iter Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Adds
ExecutionStep::AppendChildandexecute_into_builderto supportiterative execution that appends child arrays directly into builders,
avoiding intermediate materialization for chunked arrays.
Increase the max iteration limit to 2^22 from 128
part of: #7674