fix: canonicalize NaN in flat arrays_overlap float keys

[sunchao]

Why are the changes needed?

arrays_overlap answers a simple question: do two arrays contain at least one common value? For floating-point arrays, Spark treats every representation of NaN as the same value, even when their underlying IEEE 754 sign bits or payloads differ. Comet's native implementation compared those raw bit patterns instead. As a result, enabling Comet could silently change the answer to an otherwise ordinary Spark SQL query.

Consider a Parquet table containing a NaN:

CREATE TABLE t_flat(x DOUBLE) USING parquet;
INSERT INTO t_flat VALUES (DOUBLE('NaN')), (0.0D), (1.0D);

SELECT x,
       arrays_overlap(array(x), array(-x)) AS overlap
FROM t_flat;

For the first row, -x is still a NaN, but negating it at execution time changes its sign bit. Spark correctly considers that value equal to the original NaN. On Spark 3.4–4.1, before this change, Comet incorrectly treated the two representations as different:

Input x	Spark 3.4–4.1	Comet before	Comet after
NaN	true	false	true
+0.0	false	false	false
1.0	false	false	false

The +0.0 row is also a compatibility control: -x becomes -0.0, and Spark 3.4–4.1 considers the two zeros different for flat arrays. Fixing NaN comparisons must not change that result.

Spark 4.2 and later intentionally behave differently. Under SPARK-54918, Spark's optimizer normalizes both floating-point arrays_overlap operands before evaluation. Consequently, +0.0 and -0.0 overlap, and both Spark and Comet return true. The native comparison must still preserve distinct zero keys for earlier Spark versions; on Spark 4.2+, the operands have already been normalized before those keys are constructed.

On Spark 3.4–4.1, the mismatch also affects SQL null semantics. If the first array additionally contains NULL, Spark returns true because the two NaN values overlap, while the previous Comet implementation returns NULL because it misses that definite match:

SELECT arrays_overlap(
  array(x, CAST(NULL AS DOUBLE)),
  array(-x)
)
FROM t_flat
WHERE isnan(x);

-- Spark: true
-- Comet before: NULL
-- Comet after: true

Runtime negation in these examples is intentional. Parquet canonicalizes floating-point NaN values when writing them, so storing a negative or custom-payload NaN in the input file is not enough to reproduce the bug. A distinct representation must be produced after the scan.

The signed-zero boundary is therefore version-specific:

SELECT arrays_overlap(array(0.0D), array(-0.0D));

-- Spark 3.4–4.1 and Comet: false
-- Spark 4.2+ and Comet: true (operands are normalized first)

A general-purpose floating-point normalizer that also merges signed zero inside Comet's comparison key would fix the NaN mismatch but introduce a Spark 3.4–4.1 compatibility regression.

Closes #5270.

What changes were proposed in this PR?

The change gives Comet's existing flat-array comparison the same floating-point equality contract as Spark. When Comet derives a comparison key for a FLOAT or DOUBLE, every NaN representation is mapped to one canonical key. All other values retain their original bit patterns. This preserves distinct signed zeros when Spark 3.4–4.1 passes them through unchanged, while respecting Spark 4.2's earlier normalization of the operands.

Applying the correction at the shared comparison-key boundary fixes both ways the existing implementation checks for overlap: direct comparisons for small arrays and hash-based lookups for larger arrays. The execution plan, fast-path structure, null propagation, and behavior for non-floating-point values are otherwise unchanged.

This PR intentionally does not change nested-array comparisons. Spark uses a different equality contract for nested values, and the separate nested signed-zero issue is addressed in #5235.

How was this PR tested?

The native Rust regressions construct positive, negative, and signaling NaN bit patterns directly for both floating-point widths. They verify that those values overlap, that signed zeros remain distinct, that null handling remains correct, and that the same behavior holds on both sides of the small-array/hash-lookup threshold. All 23 focused arrays_overlap tests pass.

The Spark regression exercises the actual native execution path against a Parquet-backed table. It first confirms that Parquet canonicalized the stored NaN values, then uses runtime negation to produce a noncanonical representation after the scan. Spark and Comet results are compared for FLOAT and DOUBLE, both argument orders, nullable arrays, signed-zero controls, and arrays large enough to use hash lookup.

The explicit signed-zero expectations follow each Spark version's actual semantics: the two zeros remain distinct through Spark 4.1 and compare equal after Spark 4.2's operand normalization. The focused regression passes locally on Spark 3.5, Spark 4.0, and Spark 4.2.

Native formatting, compilation, and focused tests:

cd native
cargo fmt --all -- --check
cargo build
cargo test -p datafusion-comet-spark-expr --lib \
  array_funcs::arrays_overlap::tests -- --nocapture

Spark 3.5, from the repository root with JDK 17 configured:

./mvnw test -Pspark-3.5 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' \
  -Dscalastyle.skip=true

Spark 4.0, using a clean reactor build to avoid mixing Scala versions:

./mvnw clean test -Pspark-4.0 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' \
  -Dscalastyle.skip=true

Spark 4.2, including its version-aware signed-zero expectations:

./mvnw clean test -Pspark-4.2 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite floating-point NaN payloads and signed zeros' \
  -Dscalastyle.skip=true

Maven Spotless checks pass for all three locally tested Spark profiles.