fix: emit unmatched NULL-key right rows in RIGHT/FULL PiecewiseMergeJoin - #24336
Conversation
`resolve_classic_join` starts the match scan past the streamed side's NULL-keyed rows, which sort to the front under `nulls_first`. A NULL join key never matches (`NullEqualsNothing`), so for `Right`/`Full` joins those rows are unmatched and must still be emitted — but because the scan skips them, they were never recorded in `unmatched_indices` and got dropped. Record the skipped NULL-key streamed rows as unmatched for `Right`/`Full` so they are emitted with NULLs on the buffered side, matching `NestedLoopJoin`. Closes apache#24335. Co-authored-by: Claude Code
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24336 +/- ##
==========================================
+ Coverage 80.86% 81.17% +0.31%
==========================================
Files 1101 1109 +8
Lines 375446 388038 +12592
Branches 375446 388038 +12592
==========================================
+ Hits 303592 314980 +11388
- Misses 53761 54513 +752
- Partials 18093 18545 +452 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| # NULL rows are skipped by the match scan and were previously dropped instead of | ||
| # being reported as unmatched. | ||
| query II | ||
| SELECT t1.id AS left_id, t2.id AS right_id |
There was a problem hiding this comment.
is FULL join supposed to be tested as well?
There was a problem hiding this comment.
Good question — the fix path is Right | Full, so it covers FULL JOIN too. I've added a FULL JOIN test in the latest commit.
I couldn't reuse the null_join_* tables for it: on this branch's base, a FULL JOIN over those still drops the unmatched left NULL-keyed row (t1.id = NULL). That's a separate pre-existing bug — the classic Left/Full unmatched-row drop, which #23870 fixes but main doesn't have yet — unrelated to the right-side NULL fix here, so a FULL JOIN on null_join_* would fail on this base for that other reason.
To keep the test focused on what this PR fixes (the unmatched right-side NULL row), the added FULL JOIN case uses data where every left row matches, so only the right-NULL emission is exercised.
The fix applies to both `Right` and `Full`. Add a `FULL JOIN` test whose left rows all match, so it exercises only the right-side NULL-key unmatched-row emission fixed here (a separate pre-existing left-side unmatched-row drop, not addressed by this PR, would otherwise interfere). Co-authored-by: Claude Code
Address review: cast `stream_null_idx` to `u32` for the loop bound instead of casting each index inside the loop. Co-authored-by: Claude Code
|
Thanks @comphead for review! |
Which issue does this PR close?
Rationale for this change
A
RIGHT/FULLPiecewiseMergeJoinwith a range predicate drops an unmatched right-side row whose join key isNULL. ANULLkey never matches (NULL < xis UNKNOWN), so in aRIGHT/FULLjoin the row is unmatched and must still be emitted with NULLs on the left — butPiecewiseMergeJoinExecomits it, diverging fromNestedLoopJoin.Root cause:
resolve_classic_joinstarts the match scan past the streamed side'sNULL-keyed rows (they sort to the front undernulls_first). Those rows are never revisited, so forRight/Fullthey were never added tounmatched_indicesand got dropped.What changes are included in this PR?
resolve_classic_join, when skipping the streamed side's leadingNULL-key rows, record them as unmatched forRight/Fulljoins so they are emitted (with NULLs on the buffered side).Are these changes tested?
Yes.
pwmj.slt: aRIGHT JOINover the existingnull_join_*tables now emits the(NULL, NULL)row. The test fails onmain(the row is dropped) and passes with this change.NestedLoopJoin(same SQL,enable_piecewise_merge_joinon vs off): 1200 checks over randomRIGHT JOINinputs with</<=/>/>=and high right-side NULL density, 0 mismatches.Are there any user-facing changes?
RIGHT/FULLrange joins viaPiecewiseMergeJoin(behindenable_piecewise_merge_join, default off) now return unmatched right rows withNULLkeys, matchingNestedLoopJoin. No API changes.