PHOENIX-7891 Explain the query optimizer's index selection rationale

[apurtell]

Capture the query optimizer's index selection rationale with a new data model and a closed set of RULE_* and REASON_* string constants plumbed through QueryPlan/BaseQueryPlan/DelegateQueryPlan. AddPlanResult returns from the two addPlan overloads, and a DecisionState accumulator is threaded through. A new helper method assigns the winning rule and collected rejections. New ExplainPlanAttributes fields indexRule and indexRejected are set in BaseQueryPlan from getOptimizerDecision(), with matching ExplainPlanTestUtil fluent assertions indexRule, indexRuleStartsWith, indexRejectedCount, indexRejected, and indexRejectedNone.

EXPLAIN output gains INDEX <name> [<kind>] [/* <rule> */] for chosen index and one /* !INDEX <name> -- <reason> */ line per rejected index.

Co-authored-by: Claude Opus 4.8[1m] noreply@anthropic.com

[Copilot]

Pull request overview

This PR adds a structured “optimizer decision” model to capture why the query optimizer chose a particular index (or the data table), and wires that rationale through QueryPlan into both structured explain attributes and EXPLAIN text output.

Changes:

• Introduces OptimizerDecision, OptimizerReasons (RULE_* / REASON_*), and RejectedIndexEntry, and records these during plan selection in QueryOptimizer.
• Extends explain plumbing so EXPLAIN can render the chosen rule as a comment on the INDEX line and render /* !INDEX ... -- reason */ lines for rejected candidates; also exposes indexRule / indexRejected via ExplainPlanAttributes.
• Updates unit and integration tests to assert optimizer rule/rejection metadata and updated EXPLAIN output formatting.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
phoenix-core/src/test/java/org/apache/phoenix/query/QueryPlanTest.java	Updates plan assertions to include indexRule / rejected-index expectations.
phoenix-core/src/test/java/org/apache/phoenix/query/explain/ExplainPlanTestUtil.java	Adds fluent assertions for indexRule and rejected-index lists.
phoenix-core/src/test/java/org/apache/phoenix/query/explain/ExplainPlanTest.java	Updates expected EXPLAIN text/JSON and adds helper assertions for new INDEX comments and !INDEX lines.
phoenix-core/src/test/java/org/apache/phoenix/compile/TenantSpecificViewIndexCompileTest.java	Adds optimizer rule/rejection assertions to compile-time plan tests.
phoenix-core/src/test/java/org/apache/phoenix/compile/StatementHintsCompilationTest.java	Adds optimizer rule/rejection assertions for hint-related compilation paths.
phoenix-core/src/test/java/org/apache/phoenix/compile/QueryOptimizerTest.java	Asserts chosen rule and rejected index reason in optimizer selection tests.
phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java	Adds optimizer decision assertions to regression tests.
phoenix-core/src/it/java/org/apache/phoenix/end2end/TenantSpecificViewIndexIT.java	Adds optimizer decision assertions to end-to-end view/index scenarios.
phoenix-core/src/it/java/org/apache/phoenix/end2end/ReverseScanIT.java	Adds optimizer decision assertions to reverse scan scenarios.
phoenix-core/src/it/java/org/apache/phoenix/end2end/json/JsonFunctionsIT.java	Adds optimizer decision assertions for JSON/index-related plans.
phoenix-core/src/it/java/org/apache/phoenix/end2end/join/HashJoinLocalIndexIT.java	Adds optimizer decision assertions for join subplans and rejection reasons.
phoenix-core/src/it/java/org/apache/phoenix/end2end/join/HashJoinGlobalIndexIT.java	Adds optimizer decision assertions for join subplans and rejection reasons.
phoenix-core/src/it/java/org/apache/phoenix/end2end/InListIT.java	Adds optimizer decision assertions for point-lookup and range-scan IN-list plans.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ViewIndexIT.java	Adds optimizer decision assertions to view-index usage tests.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/SingleCellIndexIT.java	Asserts rule and explicit rejection reasons (e.g., NO_INDEX hint).
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/SaltedIndexIT.java	Adds optimizer decision assertions across salted index/table variants.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/PartialIndexIT.java	Adds optimizer rule assertions for partial index + hint behavior.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/MutableIndexIT.java	Adds optimizer decision assertions to mutable index selection tests.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java	Adds optimizer decision assertions and explicit local-vs-global rejection reasons.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/IndexUsageIT.java	Adds optimizer rule/rejection assertions across several index-usage patterns.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/GlobalIndexOptimizationIT.java	Adds optimizer rule/rejection assertions for optimization flows and subplans.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/GlobalIndexCheckerIT.java	Adds overload to assert expected rule and asserts rule/rejections in checks.
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/ChildViewsUseParentViewIndexIT.java	Adds optimizer rule assertions for child-view index selection.
phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java	Adds optimizer rule/rejection assertions for cost-based winner/loser decisions.
phoenix-core/src/it/java/org/apache/phoenix/end2end/BaseTenantSpecificViewIndexIT.java	Adds optimizer rule assertions to shared base IT helpers.
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/RejectedIndexEntry.java	New value type for a rejected index + reason.
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/QueryOptimizer.java	Records optimizer decisions, threads DecisionState, and tags rejection reasons.
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/OptimizerReasons.java	New closed-set string vocabulary for rules and rejection reasons.
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/OptimizerDecision.java	New model capturing chosen index, rule, and rejected index entries.
phoenix-core-client/src/main/java/org/apache/phoenix/iterate/ExplainTable.java	Renders rule comments on INDEX and prints !INDEX rejection lines.
phoenix-core-client/src/main/java/org/apache/phoenix/iterate/BaseResultIterators.java	Exposes optimizer decision to explain rendering.
phoenix-core-client/src/main/java/org/apache/phoenix/execute/DelegateQueryPlan.java	Delegates optimizer decision get/set to wrapped plan.
phoenix-core-client/src/main/java/org/apache/phoenix/execute/BaseQueryPlan.java	Stores optimizer decision and propagates it into structured explain attributes.
phoenix-core-client/src/main/java/org/apache/phoenix/compile/QueryPlan.java	Adds default get/set methods for optimizer decisions.
phoenix-core-client/src/main/java/org/apache/phoenix/compile/ExplainPlanAttributes.java	Adds indexRule and indexRejected fields + builder plumbing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[apurtell]

Test Results

Layer	Suites	Passed	Errors	Skipped	Notes
Connectionless unit tests	6	341	0	3	—
Integration tests (minicluster)	33	1204	4	14	4 pre-existing, unrelated (see below)
Total	39	1545	4	17	green except documented pre-existing flake

Unit tests

Suite	Tests	Failures	Errors	Skipped
query.explain.ExplainPlanTest	63	0	0	0
compile.QueryCompilerTest	215	0	0	2
compile.QueryOptimizerTest	49	0	0	1
compile.TenantSpecificViewIndexCompileTest	6	0	0	0
compile.StatementHintsCompilationTest	4	0	0	0
query.QueryPlanTest	4	0	0	0
Subtotal	341	0	0	3

Integration tests

Index suites

Suite	Tests	Failures	Errors	Skipped
end2end.index.GlobalIndexCheckerIT	112	0	0	6
end2end.index.MutableIndexIT	90	0	0	0
end2end.index.PartialIndexIT	60	0	0	0
end2end.index.IndexUsageIT	39	0	0	0
end2end.index.SingleCellIndexIT	14	0	0	0
end2end.index.GlobalIndexOptimizationIT	7	0	0	0
end2end.index.SaltedIndexIT	2	0	0	0
end2end.index.LocalIndexIT	60	0	4 ⚠	0

Join suites

Suite	Tests	Failures	Errors	Skipped
end2end.join.HashJoinGlobalIndexIT	33	0	0	0
end2end.join.HashJoinLocalIndexIT	34	0	0	0
end2end.join.HashJoinNoIndexIT	33	0	0	0
end2end.join.SortMergeJoinGlobalIndexIT	35	0	0	0
end2end.join.SortMergeJoinLocalIndexIT	35	0	0	0
end2end.join.SortMergeJoinNoIndexIT	35	0	0	0

Cost / optimizer

Suite	Tests	Failures	Errors	Skipped
end2end.CostBasedDecisionIT	20	0	0	0

View / tenant suites

Suite	Tests	Failures	Errors	Skipped
end2end.index.ViewIndexIT	32	0	0	2
end2end.TenantSpecificViewIndexIT (covers BaseTenantSpecificViewIndexIT)	11	0	0	0
end2end.TenantSpecificViewIndexSaltedIT	12	0	0	0
end2end.index.ChildViewsUseParentViewIndexIT	2	0	0	0
end2end.ReadOnlyViewOnReadOnlyIT	2	0	0	0

Other EXPLAIN-asserting suites

Suite	Tests	Failures	Errors	Skipped	Notes
end2end.InListIT	228	0	0	0	point lookup, data table, only candidate
end2end.ExplainPlanWithStatsEnabledIT	28	0	0	0	NO_INDEX hint paths; estimate parsing
end2end.DeleteIT	74	0	0	0	—
end2end.SequenceIT	56	0	0	0	—
end2end.Bson4IT	32	0	0	6	range scan / point lookup attributes
end2end.join.SubqueryIT	36	0	0	0	audited — data-table-only (not annotated)
end2end.join.SubqueryUsingSortMergeJoinIT	21	0	0	0	audited — data-table-only (not annotated)
end2end.UnionAllIT	19	0	0	0	—
end2end.CursorWithRowValueConstructorIT	18	0	0	0	first-line scan assertion only
end2end.json.JsonFunctionsIT	15	0	0	0	more bound PK columns, data table
end2end.QueryLoggerIT	7	0	0	0	logged vs. fresh EXPLAIN (self-consistent)
end2end.ReverseScanIT	4	0	0	0	more bound PK columns, data table
end2end.UpgradeNamespaceIT	2	0	0	0	—

Pre-existing failures (not introduced by this change)

end2end.index.LocalIndexIT reports 4 errors in testLocalIndexReverseScanShouldReturnAllRows and testLocalIndexUsedForUncoveredOrderBy (each ×isNamespaceMapped), all StackOverflowError. Tracked by PHOENIX-7893.