Skip to content

PiecewiseMergeJoin drops unmatched NULL-key right rows in RIGHT/FULL joins #24335

Description

@viirya

Describe the bug

A RIGHT/FULL PiecewiseMergeJoin with a range predicate drops an unmatched right-side row whose join key is NULL.

A NULL join key never matches under SQL semantics (NULL < x is UNKNOWN), so in a RIGHT/FULL join such a row is unmatched and must still be emitted (with NULLs on the left). PiecewiseMergeJoinExec omits it entirely.

This is independent of the LeftSemi/LeftAnti existence-join work in #23870 — it is in the classic RIGHT/FULL path and reproduces on main.

To Reproduce

enable_piecewise_merge_join on vs off (i.e. PiecewiseMergeJoin vs NestedLoopJoin) diverge:

set datafusion.optimizer.enable_piecewise_merge_join = true;

create table l(v int) as values (5);
create table r(v int) as values (10), (NULL);

select l.v, r.v from l right join r on l.v < r.v order by 1, 2;

PiecewiseMergeJoin returns:

+---+----+
| v | v  |
+---+----+
| 5 | 10 |
+---+----+

but the correct result (what NestedLoopJoin returns with the flag off) is:

+------+------+
| v    | v    |
+------+------+
| 5    | 10   |
| NULL | NULL |
+------+------+

The right row v = NULL is unmatched and should appear as (NULL, NULL).

Expected behavior

RIGHT/FULL PiecewiseMergeJoin emits every unmatched right row, including those with a NULL join key, matching NestedLoopJoin.

Additional context

Root cause: resolve_classic_join in datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs starts the match scan past the streamed side's NULL-keyed rows (they sort to the front under nulls_first). Those rows are never revisited, so for Right/Full they are dropped instead of being recorded as unmatched.

I have a fix + regression test and will open a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions