Commit 78454d9
branch-4.0: [fix](test) fix branch-4.0 P0 regression failures (build #198126) (#64525)
## What problem does this PR solve?
Stabilizes the `branch-4.0` P0 / Cloud-P0 regression pipeline. It
started from the
failures in regression build #198126 and has since grown to cover
additional flaky /
failing suites surfaced by subsequent P0 and Cloud-P0 runs.
Most commits are test-only (deflaking, isolation, session-default
pinning, branch-4.0
compatibility, or temporarily skipping cases whose feature is absent /
whose backport
is pending). Two are real product fixes backported from master, plus one
debug-point
gate change with no production impact.
### Product fixes backported from master
- **#64439** `[fix](load) fix empty statistics for forwarded INSERT` — a
forwarded
`INSERT ... SELECT` showed empty `JobDetails` (`ScannedRows=0`,
`LoadBytes=0`, …)
because the regular `Coordinator` / `CloudCoordinator` fallback path was
built without
the insert `jobId`, so `LoadManager` progress was never
initialized/updated. The
`jobId` is now preserved on both fallback paths. Fixes
`load_p0/insert/test_insert_statistic`.
- **#63666** `[fix](ann-index) Fix ANN range search state leakage and
incorrect slot
index tracking` — ANN range-search execution state was stored on the
shared `VExpr`
root, so a segment that ran ANN range search could leak state into a
segment without an
ANN index and wrongly drop the common expression; it also mixed schema
column indexes
with storage column ids. State now flows through the evaluation call and
slot status is
keyed by source column index. Adapted to 4.0 (this branch predates
`enable_result_cache`,
so that parameter is dropped from the ported signatures).
### Product change behind a debug point (no production impact)
- `[fix](compaction) let time_series_level2_file_count debug point
bypass the level2
freshness gate` — the 24h freshness gate skipped freshly-created level1
rowsets before
the `time_series_level2_file_count` debug point could collect them, so
`test_time_series_compaction_level2` never forced a level-2 merge
(tablet kept 11
rowsets instead of 2). The debug point now guards the gate; with debug
points disabled
`is_enable()` returns false, so behavior is unchanged.
### Test suites backported from master
- **#63859** `[test](regression) Add debug point ANN index-only scan
test` — replaces the
fragile `ScanBytes`-profile-based ANN suites (an unreliable index-only
proxy) with
debug-point suites that directly fail if the embedding column is read in
index-only
scenarios. Fixes `ann_index_p0/ann_index_only_scan_distance_expr` and
`ann_index_p0/ann_index_only_scan_metric_direction`.
- **partial #64238** — ports only the `scanner_profile` `actualRows`
regex fix
(`PhysicalOlapScan[<id>]` renders the numeric nereids id, not the table
name, so the
old `PhysicalOlapScan[scanner_profile]` regex could never match). The BE
CPU-time
counter rename in #64238 is intentionally NOT backported to a release
branch.
### Aligned with the `enable_insert_strict` / `enable_strict_cast`
decoupling (#63794)
- `insert_p0/insert_group_commit_into_max_filter_ratio` — cherry-pick
from master.
- `external_table_p0/hive/write/test_hive_ctas_to_doris` — set
`enable_strict_cast=true`
so the CTAS casts behave as the expected output was generated under.
### Deflaking — asynchronous profile-report races
The BE reports the detailed execution profile to FE *after* the query
result returns;
tests that fetched the profile with a fixed sleep raced that report
(empty `MergedProfile`).
- `query_profile/scanner_profile` — poll `getProfileByToken`
(Awaitility) until the scan
operator lands before asserting `MaxScanConcurrency` / backfilled
`actualRows` (sits on
top of the regex fix above; it was the only `query_profile` test with no
wait/retry).
- `query_profile/adaptive_pipeline_task_serial_read_on_limit` — same
fix: poll until
`MaxScanConcurrency` lands instead of `Thread.sleep(500)`.
- `AutoProfileTest` (FE UT) — the JMockit
`result=System.currentTimeMillis()` froze the
begin-time at record time while the duration was measured against the
real wall clock,
leaving only ~101ms of slack. Drop JMockit + both sleeps and set
`queryBeginTime`
directly (now / now-60s) for a deterministic ~60s margin.
### Deflaking — CBO / statistics timing
- `nereids_rules_p0/mv/.../partition_curd_union_rewrite` — guard the six
post-insert
`mv_rewrite_success` checks with `is_partition_statistics_ready`
(matching the suite's
own pre-mutation calls); new-partition row-count stats are reported
asynchronously, so
within the fixed sleep CBO costed the union plan above the direct plan
and emitted
"not chose".
- `nereids_syntax_p0/distribute/prune_bucket_with_bucket_shuffle_join` —
pin
`parallel_pipeline_task_num=1` so the bucket-shuffle downgrade heuristic
(`totalBucketNum < backEndNum*paraNum*0.8`) doesn't fire on a high-core
Cloud-P0 agent
and downgrade `BUCKET_SHUFFLE` → `PARTITIONED`.
- `correctness_p0/test_colocate_join_of_column_order` — inject column
stats (`set stats`)
before EXPLAIN so the COLOCATE-vs-PARTITIONED cost choice is
deterministic
(freshly-created tables have `rowCountReported=false` until the async
`CloudTabletStatMgr`
tick).
### Deflaking — cross-FE system-table semantics
- `query_p0/schema_table/test_sql_block_rule_status` — `BLOCKS` is a
cross-FE `SUM` over
each FE's non-replicated in-memory counter, so a stray counter on
another FE makes the
sum exceed 1. Set `fetch_all_fe_for_system_table=false` so the read sees
only the local
FE that ran the blocked query.
- `nereids_p0/cache/parse_sql_from_sql_cache` — drop the racy cross-FE
`assertNoCache`:
the sql-cache result lives on a shared BE chosen by a deterministic
hash, so a second FE
can legitimately serve the same query from cache. Keep the meaningful
`assertHasCache`.
### Deflaking — routine-load adaptive timeout
(`test_routine_load_adaptive_param` + `RoutineLoadTestUtils`)
The adaptive timeout is only recomputed when a task is scheduled *with*
data, and the
converged value can sit on an idle (`txnId == -1`) task; the old `txnId
!= -1` gate raced
a sub-second live-txn window.
- `RoutineLoadTestUtils.checkTaskTimeout` — skip transient `txnId == -1`
tasks and poll
until a stable task carries the expected timeout.
- restore phase — `checkTaskTimeoutWithData` feeds a small batch each
round to force the
timeout to recompute.
- increase phase — drive data each round too, and rewrite the
txn-timeout check to join on
the task UUID and read the persisted timeout from the committed
transaction
(`SHOW TRANSACTION WHERE label`) instead of catching a live txn.
### Deflaking — publish / delete visibility & file cache
- `fault_injection_p0/.../test_f_delete_publish_skip_read` — replace
`Thread.sleep(12000)`
with a bounded Awaitility poll until the delete (v4) is actually
published and visible
(`k1=2` gone). The storage engine applies the delete correctly; the race
was in publish
timing (the `disable_block` request could queue behind the in-flight
stream load).
- `external_table_p0/cache/test_file_cache_statistics` (and
`test_file_cache_query_limit`)
— `file_cache_statistics` reports one row per `(cache_path, metric)` and
a data file
hashes to exactly one path, so a bare `limit 1` inspected an arbitrary
path. Sum each
metric across paths (`SUM(CAST(METRIC_VALUE AS DOUBLE))`) and replace
fixed sleeps with
Awaitility.
### Determinism — pin fuzzed session defaults / fix drifted expected
output
- `check_before_quit` — pin the two fuzzed variant session defaults to 0
so the
CREATE → recreate → show-create round-trip is idempotent for
property-less variant columns.
- `variant_p0/.../test_variant_compaction_with_sparse_limit` — pin
`default_variant_max_subcolumns_count=2048` (the FE default the `.out`
was generated
under) so nested paths stay typed subcolumns.
- `audit/test_audit_log_behavior` — correct the drifted `query_id`
column to `varchar(128)`
(`Config.label_regex_length`); `stmt_type` stays `varchar(48)`.
### branch-4.0 compatibility — remove unsupported
`enable_condition_cache` variable
`enable_condition_cache` does not exist on branch-4.0, so `set
enable_condition_cache=false`
fails with "Unknown system variable". Removing the line is a semantic
no-op (the branch is
already in the cache-disabled state the tests expect). Applied to the
`ann_index_only_scan`
debug-point cases (from #63859) and the `ann_range_search` cases (from
#63666).
### Test isolation
- `backup_restore/test_backup_restore_reset_index_id` (+
`test_backup_restore_inverted_idx`,
`restore_p0/test_validate_restore_inverted_idx`) — these three suites
mutate the
process-global `restore_reset_index_id` FE config; running in parallel,
one could flip it
between another suite's BACKUP and RESTORE (build #198126). They are now
in the
`nonConcurrent` group so they run serially. The FE reset logic itself is
correct; purely
a test-isolation fix.
### Temporarily skipped / disabled
- `temp_table_p0/test_temp_table` — **skipped** (the temp-table feature
is no longer
maintained). An earlier commit had hardened its replica-count detection
for build
#198126, but the suite is now skipped outright.
- `ann_index_p0/ann_range_search_pushdown_regression` — disabled pending
backport of
**#64082**: it builds an under-sized IVF scan that relies on the BE
skipping ANN index
build for segments smaller than `nlist`; without #64082 the single-row
INSERT fails at
segment finalize with faiss `nx >= k`. Re-enable after picking #64082.
- `external_table_p0/hive/test_parquet_join_runtime_filter` — skipped
(feature not
supported on 4.0).
- `shape_check/tpcds_sf100/shape/query64` and
`shape_check/tpcds_sf1000/shape/query64` —
ignored (unstable shape checks).
## Release note
None
## Check List (For Committer)
- [x] Test <!-- regression tests -->
## Check List (For Reviewer who provide ASF feature)
- [ ] Performance regression
---------
Co-authored-by: hui lai <laihui@selectdb.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: meiyi <meiyi@selectdb.com>1 parent bc17fb7 commit 78454d9
49 files changed
Lines changed: 1449 additions & 1175 deletions
File tree
- be
- src
- olap
- rowset/segment_v2
- vec/exprs
- test/olap/vector_search
- fe/fe-core/src
- main/java/org/apache/doris
- catalog
- cloud
- catalog
- qe
- qe
- test/java/org/apache/doris/common/profile
- regression-test
- data
- ann_index_p0
- audit
- framework/src/main/groovy/org/apache/doris/regression/util
- suites
- ann_index_p0
- backup_restore
- check_before_quit
- correctness_p0
- external_table_p0
- cache
- hive
- write
- fault_injection_p0/flexible/legacy
- insert_p0
- load_p0/routine_load
- nereids_p0/cache
- nereids_rules_p0/mv/union_rewrite
- nereids_syntax_p0/distribute
- query_p0/schema_table
- query_profile
- restore_p0
- shape_check
- tpcds_sf1000/shape
- tpcds_sf100/shape
- temp_table_p0
- variant_p0/predefine
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| |||
378 | 379 | | |
379 | 380 | | |
380 | 381 | | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
381 | 387 | | |
382 | | - | |
| 388 | + | |
| 389 | + | |
383 | 390 | | |
384 | 391 | | |
385 | 392 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1110 | 1110 | | |
1111 | 1111 | | |
1112 | 1112 | | |
| 1113 | + | |
1113 | 1114 | | |
1114 | 1115 | | |
1115 | | - | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
1116 | 1121 | | |
1117 | 1122 | | |
1118 | 1123 | | |
| |||
1122 | 1127 | | |
1123 | 1128 | | |
1124 | 1129 | | |
1125 | | - | |
1126 | | - | |
1127 | | - | |
1128 | | - | |
1129 | | - | |
1130 | | - | |
1131 | | - | |
1132 | | - | |
1133 | | - | |
1134 | | - | |
1135 | 1130 | | |
1136 | 1131 | | |
1137 | 1132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
534 | 534 | | |
535 | 535 | | |
536 | 536 | | |
537 | | - | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
538 | 540 | | |
539 | 541 | | |
540 | 542 | | |
| |||
543 | 545 | | |
544 | 546 | | |
545 | 547 | | |
546 | | - | |
547 | | - | |
| 548 | + | |
| 549 | + | |
548 | 550 | | |
549 | 551 | | |
550 | 552 | | |
| |||
611 | 613 | | |
612 | 614 | | |
613 | 615 | | |
| 616 | + | |
614 | 617 | | |
615 | 618 | | |
616 | 619 | | |
| |||
634 | 637 | | |
635 | 638 | | |
636 | 639 | | |
637 | | - | |
| 640 | + | |
638 | 641 | | |
639 | 642 | | |
640 | 643 | | |
| |||
648 | 651 | | |
649 | 652 | | |
650 | 653 | | |
651 | | - | |
652 | 654 | | |
653 | 655 | | |
654 | 656 | | |
655 | | - | |
| 657 | + | |
656 | 658 | | |
657 | 659 | | |
658 | | - | |
| 660 | + | |
| 661 | + | |
659 | 662 | | |
660 | 663 | | |
661 | | - | |
| 664 | + | |
662 | 665 | | |
663 | 666 | | |
664 | 667 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
94 | | - | |
| 94 | + | |
| 95 | + | |
95 | 96 | | |
96 | 97 | | |
97 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
991 | 991 | | |
992 | 992 | | |
993 | 993 | | |
994 | | - | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
995 | 997 | | |
996 | 998 | | |
997 | 999 | | |
| |||
1009 | 1011 | | |
1010 | 1012 | | |
1011 | 1013 | | |
1012 | | - | |
1013 | | - | |
1014 | | - | |
1015 | | - | |
1016 | | - | |
1017 | | - | |
1018 | | - | |
1019 | | - | |
1020 | 1014 | | |
1021 | 1015 | | |
1022 | 1016 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
87 | 96 | | |
88 | 97 | | |
89 | 98 | | |
| |||
342 | 351 | | |
343 | 352 | | |
344 | 353 | | |
345 | | - | |
| 354 | + | |
| 355 | + | |
346 | 356 | | |
347 | 357 | | |
348 | 358 | | |
| |||
352 | 362 | | |
353 | 363 | | |
354 | 364 | | |
355 | | - | |
356 | | - | |
357 | | - | |
358 | | - | |
359 | 365 | | |
360 | 366 | | |
361 | 367 | | |
| |||
436 | 442 | | |
437 | 443 | | |
438 | 444 | | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
443 | | - | |
444 | | - | |
445 | | - | |
446 | 445 | | |
447 | 446 | | |
448 | 447 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
433 | 433 | | |
434 | 434 | | |
435 | 435 | | |
436 | | - | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
437 | 441 | | |
438 | 442 | | |
439 | 443 | | |
440 | 444 | | |
| 445 | + | |
441 | 446 | | |
442 | 447 | | |
443 | | - | |
| 448 | + | |
444 | 449 | | |
445 | | - | |
| 450 | + | |
446 | 451 | | |
447 | 452 | | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
448 | 462 | | |
449 | | - | |
| 463 | + | |
450 | 464 | | |
451 | 465 | | |
452 | 466 | | |
453 | 467 | | |
454 | | - | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
455 | 471 | | |
456 | 472 | | |
457 | 473 | | |
458 | 474 | | |
459 | 475 | | |
460 | | - | |
461 | | - | |
| 476 | + | |
| 477 | + | |
462 | 478 | | |
463 | 479 | | |
464 | | - | |
465 | | - | |
| 480 | + | |
| 481 | + | |
466 | 482 | | |
467 | 483 | | |
468 | 484 | | |
469 | 485 | | |
470 | | - | |
| 486 | + | |
| 487 | + | |
471 | 488 | | |
472 | 489 | | |
473 | 490 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
358 | 358 | | |
359 | 359 | | |
360 | 360 | | |
361 | | - | |
| 361 | + | |
| 362 | + | |
362 | 363 | | |
363 | 364 | | |
364 | 365 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
234 | 234 | | |
235 | 235 | | |
236 | 236 | | |
237 | | - | |
| 237 | + | |
| 238 | + | |
238 | 239 | | |
239 | 240 | | |
240 | | - | |
241 | | - | |
242 | | - | |
| 241 | + | |
243 | 242 | | |
244 | 243 | | |
245 | 244 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
110 | | - | |
| 110 | + | |
| 111 | + | |
111 | 112 | | |
112 | 113 | | |
113 | 114 | | |
| |||
0 commit comments