Commit 0c5c97b
authored
fix(functions-aggregate): drain CORR state vectors for streaming aggregation (#19669)
## Which issue does this PR close?
- N/A
## Rationale for this change
This change addresses a failure in the `CORR` aggregate function when
running in streaming mode. The `CorrelationGroupsAccumulator`
(introduced in [PR
#13581](#13581)) was failing to
drain its state vectors during `EmitTo::First` calls, causing internal
state to persist across emissions. This led to memory leaks, incorrect
results for subsequent groups, and "length mismatch" errors because the
internal vector sizes diverged from the number of emitted groups.
### Reproducer
```sql
# Setup data
CREATE TABLE stream_test (
g INT,
x DOUBLE,
y DOUBLE
) AS VALUES
(1, 1.0, 1.0), (1, 2.0, 2.0),
(2, 1.0, 5.0), (2, 2.0, 5.0),
(3, 1.0, 1.0), (3, 2.0, 2.0);
# Trigger streaming aggregation via sorted subquery
SELECT
g,
CORR(x, y)
FROM (SELECT * FROM stream_test ORDER BY g LIMIT 10000)
GROUP BY g
ORDER BY g;
```
**Before**: `DataFusion error: Arrow error: Invalid argument error: all
columns in a record batch must have the same length`
**After**:
```
1 1
2 NULL
3 1
```
## What changes are included in this PR?
This PR is structured into two commits: the first adds a failing test
case to demonstrate the issue, and the second implements the fix.
The accumulator now uses `emit_to.take_needed()` in both `evaluate` and
`state` to properly consume the emitted portions of the state vectors.
Additionally, the `size()` implementation has been updated to use vector
capacity for more accurate memory accounting.
## Are these changes tested?
Yes, a new test case in `aggregate.slt` triggers streaming aggregation
via an ordered subquery. This test previously crashed with an Arrow
length mismatch error and now produces correct results.
## Are there any user-facing changes?
Yes, SQL queries that trigger streaming aggregation using `CORR`
(typically those with specific ordering requirements) will now succeed
instead of failing with a length mismatch error.1 parent 9fa7500 commit 0c5c97b
2 files changed
Lines changed: 315 additions & 28 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
411 | 411 | | |
412 | 412 | | |
413 | 413 | | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
419 | 423 | | |
420 | 424 | | |
421 | 425 | | |
| |||
427 | 431 | | |
428 | 432 | | |
429 | 433 | | |
430 | | - | |
431 | 434 | | |
432 | | - | |
433 | | - | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
438 | 441 | | |
439 | 442 | | |
440 | 443 | | |
| |||
470 | 473 | | |
471 | 474 | | |
472 | 475 | | |
473 | | - | |
474 | | - | |
475 | | - | |
476 | | - | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
477 | 483 | | |
478 | 484 | | |
479 | | - | |
480 | | - | |
481 | | - | |
482 | | - | |
483 | | - | |
484 | | - | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
485 | 491 | | |
486 | 492 | | |
487 | 493 | | |
| |||
537 | 543 | | |
538 | 544 | | |
539 | 545 | | |
540 | | - | |
541 | | - | |
542 | | - | |
543 | | - | |
544 | | - | |
545 | | - | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
546 | 552 | | |
547 | 553 | | |
548 | 554 | | |
| |||
0 commit comments