[fix](partition_prune) Move the pruning of predicates that are always true after partition pruning into the PlanPostProcessor#63111
Open
feiniaofeiafei wants to merge 6 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29511 ms |
Contributor
TPC-DS: Total hot run time: 170534 ms |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29722 ms |
Contributor
TPC-DS: Total hot run time: 170733 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run p0 |
Contributor
|
I agree with the core idea: partition-prunable predicates should not be removed before MV rewrite, otherwise MV matching may see an incomplete query predicate set and produce an unsafe rewrite. One concern is where the deferred-removal state is stored. The current implementation records
Putting that proof in A cleaner structure may be to attach this information to the scan itself, e.g. class PartitionPruneInfo {
Set<Long> provedPartitionIds;
List<Slot> partitionSlots;
Set<Expression> prunableConjuncts;
}Then the post processor can simply inspect scan.getPartitionPruneInfo() and remove a conjunct only when: partitionPruneInfo.getProvedPartitionIds().containsAll(scan.getSelectedPartitionIds())This keeps the invariant local to the scan and avoids using StatementContext as a side table for plan-node-specific proof. |
…nstead of StatementContext ### What problem does this PR solve? Problem Summary: Move PartitionPrunablePredicate info from StatementContext (keyed by TableIdentifier) onto LogicalOlapScan / PhysicalOlapScan so it follows the scan through with*/deepCopy and is included in equals. Also propagate the info onto the MV scan rewritten by SyncMaterializationContext#getScanPlan so the post-processor can still strip the deferred conjuncts after sync MV rewrite. ### Release note None ### Check List (For Author) - Test: Regression test (prune_predicates_mv_test.groovy) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
One olap scan can carry at most one PartitionPrunablePredicate, so use Optional instead of Set. Simplifies merge in PruneOlapScanPartition and iteration in PrunePartitionPredicate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
feiniaofeiafei
force-pushed
the
fix/cir-20138-mv-partition-predicate-compensate
branch
from
d817a84 to
4c371b5
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29486 ms |
Contributor
TPC-DS: Total hot run time: 169360 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run feut |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #57169
Problem Summary:
After partition pruning, predicates that are evaluated to constant true are removed as an optimization (introduced by #57169). However, this introduced a bug in materialized view rewriting: when such predicates are removed, they are not compensated above the rewritten materialized view. Since the materialized view itself is not partitioned, this leads to incorrect query results.
This PR fixes the issue by moving the removal of constant-true predicates to the planPostProcessors phase. This ensures that the predicate removal does not interfere with materialized view rewriting, preserving correctness while still retaining the optimization benefit.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)