Describe the bug
A RIGHT/FULL PiecewiseMergeJoin with a range predicate drops an unmatched right-side row whose join key is NULL.
A NULL join key never matches under SQL semantics (NULL < x is UNKNOWN), so in a RIGHT/FULL join such a row is unmatched and must still be emitted (with NULLs on the left). PiecewiseMergeJoinExec omits it entirely.
This is independent of the LeftSemi/LeftAnti existence-join work in #23870 — it is in the classic RIGHT/FULL path and reproduces on main.
To Reproduce
enable_piecewise_merge_join on vs off (i.e. PiecewiseMergeJoin vs NestedLoopJoin) diverge:
set datafusion.optimizer.enable_piecewise_merge_join = true;
create table l(v int) as values (5);
create table r(v int) as values (10), (NULL);
select l.v, r.v from l right join r on l.v < r.v order by 1, 2;
PiecewiseMergeJoin returns:
+---+----+
| v | v |
+---+----+
| 5 | 10 |
+---+----+
but the correct result (what NestedLoopJoin returns with the flag off) is:
+------+------+
| v | v |
+------+------+
| 5 | 10 |
| NULL | NULL |
+------+------+
The right row v = NULL is unmatched and should appear as (NULL, NULL).
Expected behavior
RIGHT/FULL PiecewiseMergeJoin emits every unmatched right row, including those with a NULL join key, matching NestedLoopJoin.
Additional context
Root cause: resolve_classic_join in datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs starts the match scan past the streamed side's NULL-keyed rows (they sort to the front under nulls_first). Those rows are never revisited, so for Right/Full they are dropped instead of being recorded as unmatched.
I have a fix + regression test and will open a PR.
Describe the bug
A
RIGHT/FULLPiecewiseMergeJoinwith a range predicate drops an unmatched right-side row whose join key isNULL.A
NULLjoin key never matches under SQL semantics (NULL < xis UNKNOWN), so in aRIGHT/FULLjoin such a row is unmatched and must still be emitted (with NULLs on the left).PiecewiseMergeJoinExecomits it entirely.This is independent of the LeftSemi/LeftAnti existence-join work in #23870 — it is in the classic
RIGHT/FULLpath and reproduces onmain.To Reproduce
enable_piecewise_merge_joinon vs off (i.e.PiecewiseMergeJoinvsNestedLoopJoin) diverge:PiecewiseMergeJoin returns:
but the correct result (what
NestedLoopJoinreturns with the flag off) is:The right row
v = NULLis unmatched and should appear as(NULL, NULL).Expected behavior
RIGHT/FULLPiecewiseMergeJoinemits every unmatched right row, including those with aNULLjoin key, matchingNestedLoopJoin.Additional context
Root cause:
resolve_classic_joinindatafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rsstarts the match scan past the streamed side'sNULL-keyed rows (they sort to the front undernulls_first). Those rows are never revisited, so forRight/Fullthey are dropped instead of being recorded as unmatched.I have a fix + regression test and will open a PR.