[SPARK-58384][SQL] Preserve null semantics in OptimizeJoinCondition

[MrHappyEnding]

What changes were proposed in this pull request?

Restrict OptimizeJoinCondition to rewriting null-safe equality patterns only at the root of a join condition or beneath AND/OR.

The rule no longer performs this rewrite beneath NOT, where the difference between NULL and FALSE is observable.

Why are the changes needed?

When exactly one operand is NULL, the original pattern returns NULL, while <=> returns FALSE.

This difference does not matter at the root of a join condition, but under NOT it can cause Spark to incorrectly keep extra rows.

Does this PR introduce any user-facing change?

Yes. Join conditions containing this pattern beneath NOT now return the correct rows.

For the SPARK-58384 reproduction, the result changes from four rows to only:

[0,10,1,22]

How was this patch tested?

Added tests covering:

• No rewrite beneath NOT.
• Continued rewriting beneath AND and OR.
• The end-to-end join result with null values.

Ran:

build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizeJoinConditionSuite'
build/sbt 'sql/testOnly org.apache.spark.sql.DataFrameJoinSuite -- -z "SPARK-58384"'
build/sbt 'catalyst/scalastyle' 'catalyst/Test/scalastyle' 'sql/Test/scalastyle'

Was this patch authored or co-authored using generative AI tooling?

co-authored by Codex (GPT-5)

[uros-b]

Thank you @MrHappyEnding! cc @cloud-fan who merged the original OptimizeJoinCondition rule

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits.
The semantic fix and its tests are sound; one non-blocking optimizer-allocation regression should be tightened.

Suggestions (1)

• sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJoinCondition.scala:48: Preserve tree-pattern pruning and unchanged Boolean nodes so unrelated join conditions are not traversed and rebuilt. -- see inline

Verification

I traced all truth-value cases for the rewrite. Both forms agree for equal values, unequal non-null values, and two nulls; with exactly one null, the rewrite changes NULL to FALSE. At a join root and through AND/OR this cannot create a new TRUE result, and the helper's catch-all prevents the unsafe propagation beneath NOT.

[cloud-fan]

Please keep the OR tree-pattern guard and return the original AND/OR node when neither child changes. Without those checks, this rule now traverses and allocates a fresh Boolean tree for every unrelated join condition that the previous transformWithPruning skipped.

[MrHappyEnding]

Thanks! Updated the traversal to preserve tree-pattern pruning and reuse unchanged AND/OR nodes.

[cloud-fan]

1 addressed, 0 remaining, 0 new.
0 blocking, 0 non-blocking, 0 nits.
The semantic fix, traversal boundaries, prior performance concern, and test coverage are sound at the current head.

Verification

I traced the rewrite across all operand states. The forms agree for equal values, unequal non-null values, and two NULLs; with exactly one NULL, NULL becomes FALSE. At a join root or beneath only AND/OR, that change cannot create or remove a TRUE result, and the catch-all prevents propagation beneath NOT. I also verified that containsPattern is a constant-time lookup in the cached tree-pattern bit set, so the recursive pruning checks do not rescan subtrees.

[cloud-fan]

Merge Summary:

• merged into master cea566b
• merged into branch-4.x 5db7254
• merged into branch-4.3 f67d154

Posted by merge_spark_pr.py