Skip to content

[core] Fix dropping stats in data evolution scan - #8994

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix/data-evolution-drop-stats
Draft

[core] Fix dropping stats in data evolution scan#8994
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:fix/data-evolution-drop-stats

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

DataEvolutionFileStoreScan keeps file statistics during manifest reading because predicate evaluation and per-file column pruning are performed in postFilterManifestEntries.

When neither an input filter nor a read type is configured, postFilterManifestEntriesEnabled() previously returned false. As a result, calling dropStats() had no effect because the post-filtering hook that removes statistics was skipped.

This change:

  • always enables post-filtering for data-evolution scans;
  • preserves row-id grouping, statistics filtering, and read-type pruning when an input filter or read type is present;
  • directly drops statistics when neither an input filter nor a read type is present;
  • adds a regression test for dropStats() without a filter or read type.

Tests

  • mvn -pl paimon-core -DwildcardSuites=none -Dtest=DataEvolutionTableTest#testDropStatsWithoutFilterOrReadType+testSystemFieldOnlyProjectionIsNotPruned test
  • mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=DataEvolutionTableTest test (42 tests)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant