fix: block inverted index V1 creation in FE [DORIS-26415]#64522
Open
LIANG751234313 wants to merge 1 commit into
Open
fix: block inverted index V1 creation in FE [DORIS-26415]#64522LIANG751234313 wants to merge 1 commit into
LIANG751234313 wants to merge 1 commit into
Conversation
Explicitly specifying inverted index V1 format now returns an error to prevent new tables from using V1. V1 does not support small file merging and bypasses the packed file path, causing excessive PUT requests to object storage (~300-400/s observed, 70-80% from inverted V1). Error: Inverted index V1 is deprecated and no longer allowed for new index creation. Please use inverted index V2. Existing V1 tables remain compatible, no forced migration.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #DORIS-26415
Problem
When users copy historical DDL statements that explicitly set inverted index format to V1, new tables are silently created with V1. Inverted index V1 does not support small file merging and bypasses the packed file path, significantly amplifying PUT requests to object storage.
Observed impact: clusters with high write QPS are seeing300–400 PUT/s, with 70–80% attributed to inverted index V1.
Solution
Block new inverted index V1 creation at the FE level:
inverted_index_storage_format = v1on a new indexnow returns an error immediately.are unaffected. No forced migration.
Error message:Inverted index V1 is deprecated and no longer allowed for new index creation.Please use inverted index V2.
Coverage
CREATE TABLEwith inline index definitionALTER TABLE ADD INDEXChecklist