Skip to content

Commit 0a429a3

Browse files
authored
fix: emit unmatched NULL-key right rows in RIGHT/FULL PiecewiseMergeJoin (#24336)
## Which issue does this PR close? - Closes #24335. ## Rationale for this change A `RIGHT`/`FULL` `PiecewiseMergeJoin` with a range predicate drops an unmatched right-side row whose join key is `NULL`. A `NULL` key never matches (`NULL < x` is UNKNOWN), so in a `RIGHT`/`FULL` join the row is unmatched and must still be emitted with NULLs on the left — but `PiecewiseMergeJoinExec` omits it, diverging from `NestedLoopJoin`. ```sql create table l(v int) as values (5); create table r(v int) as values (10), (NULL); select l.v, r.v from l right join r on l.v < r.v; -- drops (NULL, NULL) ``` Root cause: `resolve_classic_join` starts the match scan past the streamed side's `NULL`-keyed rows (they sort to the front under `nulls_first`). Those rows are never revisited, so for `Right`/`Full` they were never added to `unmatched_indices` and got dropped. ## What changes are included in this PR? - In `resolve_classic_join`, when skipping the streamed side's leading `NULL`-key rows, record them as unmatched for `Right`/`Full` joins so they are emitted (with NULLs on the buffered side). ## Are these changes tested? Yes. - Regression test in `pwmj.slt`: a `RIGHT JOIN` over the existing `null_join_*` tables now emits the `(NULL, NULL)` row. The test fails on `main` (the row is dropped) and passes with this change. - Verified more broadly with a differential fuzz against `NestedLoopJoin` (same SQL, `enable_piecewise_merge_join` on vs off): 1200 checks over random `RIGHT JOIN` inputs with `<`/`<=`/`>`/`>=` and high right-side NULL density, 0 mismatches. ## Are there any user-facing changes? `RIGHT`/`FULL` range joins via `PiecewiseMergeJoin` (behind `enable_piecewise_merge_join`, default off) now return unmatched right rows with `NULL` keys, matching `NestedLoopJoin`. No API changes.
1 parent e76f1af commit 0a429a3

2 files changed

Lines changed: 55 additions & 0 deletions

File tree

datafusion/physical-plan/src/joins/piecewise_merge_join/classic_join.rs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,6 +481,16 @@ fn resolve_classic_join(
481481
buffer_idx = buffered_null_idx;
482482
stream_idx = stream_null_idx;
483483
batch_process_state.processed_null_count = true;
484+
485+
// The scan below starts past the streamed side's NULL-keyed rows, which
486+
// sit at the front (`nulls_first`). A NULL join key never matches under
487+
// `NullEqualsNothing`, so for `Right`/`Full` those rows are unmatched and
488+
// must still be emitted; record them here since the scan will skip them.
489+
if matches!(join_type, JoinType::Right | JoinType::Full) {
490+
for row_idx in 0..stream_null_idx as u32 {
491+
batch_process_state.unmatched_indices.append_value(row_idx);
492+
}
493+
}
484494
}
485495

486496
// Our buffer_idx variable allows us to start probing on the buffered side where we last matched

datafusion/sqllogictest/test_files/pwmj.slt

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,5 +342,50 @@ ORDER BY 1,2;
342342
1 3
343343
2 3
344344

345+
# RIGHT JOIN: every right row must be emitted, including one whose join key is
346+
# NULL. A NULL key never matches (NullEqualsNothing), so t2.id = NULL is
347+
# unmatched and must appear as (NULL, NULL). Regression test: the streamed-side
348+
# NULL rows are skipped by the match scan and were previously dropped instead of
349+
# being reported as unmatched.
350+
query II
351+
SELECT t1.id AS left_id, t2.id AS right_id
352+
FROM null_join_t1 t1
353+
RIGHT JOIN null_join_t2 t2
354+
ON t1.id < t2.id
355+
ORDER BY 1,2;
356+
----
357+
1 3
358+
2 3
359+
NULL 1
360+
NULL NULL
361+
362+
# FULL JOIN: the same fix applies to the FULL path. An unmatched right row with
363+
# a NULL key must still be emitted as (NULL, NULL). This uses data where every
364+
# left row matches, so only the right-side NULL emission fixed here is
365+
# exercised (the separate left-side unmatched-row drop, tracked elsewhere, is
366+
# not).
367+
statement ok
368+
CREATE TABLE full_null_t1 (id INT);
369+
370+
statement ok
371+
CREATE TABLE full_null_t2 (id INT);
372+
373+
statement ok
374+
INSERT INTO full_null_t1 VALUES (10), (20);
375+
376+
statement ok
377+
INSERT INTO full_null_t2 VALUES (100), (NULL);
378+
379+
query II
380+
SELECT t1.id AS left_id, t2.id AS right_id
381+
FROM full_null_t1 t1
382+
FULL JOIN full_null_t2 t2
383+
ON t1.id < t2.id
384+
ORDER BY 1,2;
385+
----
386+
10 100
387+
20 100
388+
NULL NULL
389+
345390
statement ok
346391
set datafusion.optimizer.enable_piecewise_merge_join = false;

0 commit comments

Comments
 (0)