[fix](runtime-filter) Backport runtime filter fixes to branch-4.0#63083
Merged
yiguolei merged 9 commits intoMay 11, 2026
Conversation
…path ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#59786 Problem Summary: PR apache#59786 (commit e78d089) refactored Scanner::try_append_late_arrival_runtime_filter() and accidentally removed the trailing `_applied_rf_num = arrived_rf_num;` assignment. As a result, _applied_rf_num is permanently 0 after construction: * the fast-path `_applied_rf_num == _total_rf_num` early return at the top of the function never fires, so every batch goes through the full late-arrival check; * `arrived_rf_num == _applied_rf_num` only short-circuits when no runtime filter has ever arrived, so once any RF arrives every subsequent call needlessly clears _conjuncts, re-clones them, and appends the old ctxs into _stale_expr_ctxs (CPU waste + slow memory growth); * the `ApplyAllRuntimeFilters=True` info string in profile (file_scanner.cpp) is never emitted; * `DCHECK(_applied_rf_num < _total_rf_num)` is effectively dead because the left-hand side is always 0. Restore the single missing assignment after cloning the new conjunct ctxs to bring back the original behavior. ### Release note None ### Check List (For Author) - Test: No need to test (one-line restoration of removed assignment; behavior covered by existing runtime-filter regression tests) - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 884c978)
Related PR: apache#62872 Problem Summary: Lock in the fix for `Scanner::try_append_late_arrival_runtime_filter` so the regression introduced in PR apache#59786 cannot reappear. The new BE unit test instantiates a Scanner over MockScanLocalState/MockScanOperatorX, signals two runtime filters, and asserts that `_applied_rf_num` advances to `_total_rf_num` and that the second invocation hits the fast-path early return without re-cloning conjuncts. None - Test: Unit Test - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 675dee1)
…g VExprContext recreation
Issue Number: close #xxx
Problem Summary:
RuntimeFilter selectivity tracking was completely non-functional because
`sampling_frequency` was lost during VExprContext recreation.
In `RuntimeFilterConsumer::_get_push_exprs()`, `sampling_frequency=32` was set
on a temporary `probe_ctx` VExprContext. However, only VRuntimeFilterWrapper
expressions (VExpr) were returned, not the VExprContext. When
`_append_rf_into_conjuncts()` later created a new VExprContext via
`VExprContext::create_shared(expr)`, the new context had default
`_sampling_frequency=-1` (DISABLE_SAMPLING).
With `_sampling_frequency=-1`, the condition `(_judge_counter++) >= -1`
evaluated to `0 >= -1 → true` on every call, causing `reset_judge_selectivity()`
to fire every time. This meant selectivity counters were perpetually reset
and never accumulated, making the runtime filter selectivity optimization
completely ineffective.
The fix stores `sampling_frequency` in VRuntimeFilterWrapper (which survives
VExprContext recreation) and propagates it to VExprContext in
`VRuntimeFilterWrapper::open()`, which is called on both original and cloned
contexts.
Fixed a bug where RuntimeFilter selectivity tracking was non-functional due to
sampling_frequency being lost during VExprContext recreation, causing runtime
filters that should be skipped (due to low selectivity) to never be identified.
- Test: Unit Test
- Added 2 regression tests to runtime_filter_selectivity_test.cpp
- Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp
- All 22 tests pass
- Behavior changed: No (selectivity tracking was broken before, this makes it work as designed)
- Does this need documentation: No
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit f741922)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> (cherry picked from commit 69e6004)
…ing test StubVExpr (leaf node with no children) defaulted to is_constant()=true via VExpr::is_constant()'s all_of on empty children. On second open() call (context2), _constant_col was already populated from context1, causing DCHECK(column_wrapper != nullptr) to fail in get_const_col() when called with nullptr from VExpr::open(). Fix: override is_constant() to return false, matching SLOT_REF semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 641c010)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### What problem does this PR solve? Issue Number: close apache#62872 Related PR: apache#62872 Problem Summary: Adapt the late-arrival runtime filter scanner unit test to branch-4.0 namespaces by qualifying DataTypeFactory. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=ScannerLateArrivalRfTest.*:VRuntimeFilterWrapperSamplingTest.*:RuntimeFilterSelectivityTest.* -j 16 - Behavior changed: No - Does this need documentation: No Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Backports two BE runtime-filter fixes into branch-4.0: (1) correctly advancing the scanner’s applied-RF counter in the late-arrival path, and (2) preserving runtime-filter selectivity sampling_frequency across VExprContext recreation by storing it on VRuntimeFilterWrapper and reapplying it during open().
Changes:
- Restore
_applied_rf_num = arrived_rf_numafter late-arrival RF conjunct re-cloning inScanner::try_append_late_arrival_runtime_filter(). - Persist and propagate runtime-filter selectivity
sampling_frequencyby adding it toVRuntimeFilterWrapperand setting it on the (recreated)VExprContextduringopen(). - Add/extend BE unit tests covering late-arrival RF counter advancement and selectivity sampling behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| be/src/vec/exec/scan/scanner.cpp | Restores _applied_rf_num update so late-arrival RF application state advances correctly. |
| be/src/runtime_filter/runtime_filter_consumer.cpp | Computes sampling frequency once and passes it into VRuntimeFilterWrapper so it survives context recreation. |
| be/src/vec/exprs/vruntimefilter_wrapper.h | Extends wrapper to store sampling_frequency (defaulting to DISABLE) in the expression object. |
| be/src/vec/exprs/vruntimefilter_wrapper.cpp | Propagates stored sampling_frequency to the active VExprContext in open(). |
| be/test/vec/exec/scanner_late_arrival_rf_test.cpp | New regression test verifying late-arrival path advances _applied_rf_num and fast-path short-circuits on subsequent calls. |
| be/test/runtime_filter/vruntimefilter_wrapper_sampling_test.cpp | New tests validating sampling_frequency propagation and survival across VExprContext recreation. |
| be/test/runtime_filter/runtime_filter_selectivity_test.cpp | Adds regression tests around default/valid sampling frequencies and reset/accumulation behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| SlotDescriptor slot_desc; | ||
| slot_desc._type = vectorized::DataTypeFactory::instance().create_data_type( | ||
| PrimitiveType::TYPE_INT, false); | ||
| TupleDescriptor tuple_desc; |
Contributor
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #62872, #62355
Problem Summary: Backport the branch-4.0 runtime filter fixes that are still missing from branch-4.0:
_applied_rf_num = arrived_rf_numin the late-arrival runtime filter scanner path, so applied runtime filter count advances correctly.sampling_frequencyacrossVRuntimeFilterWrapper/VExprContextrecreation.During the branch selection check, #62854 was not needed for branch-4.0/branch-4.1 because the affected
length()OFFSET access-path optimization is absent there, and #60952 was already present on both branches.Release note
None
Check List (For Author)
./build.sh --be./run-be-ut.sh --run --filter=ScannerLateArrivalRfTest.*:VRuntimeFilterWrapperSamplingTest.*:RuntimeFilterSelectivityTest.* -j 16build-support/check-format.shbuild-support/run-clang-tidy.sh --base upstream/branch-4.0 --build-dir be/ut_build_ASANvia the current script from master, because branch-4.0 does not contain this helper script