feat: full native make_interval with 2.6x faster than spark and 1.1x faster than codegen dispatch

[peterxcli]

Which issue does this PR close?

Closes #5279.
Closes #5131.

Rationale for this change

Comet represented Spark CalendarIntervalType as Arrow IntervalMonthDayNano. Converting Spark's microseconds to nanoseconds reduced the valid elapsed-time range by 1,000x, while the native datafusion-spark kernel also coerced Decimal(18,6) seconds to Float64, losing microsecond precision.

Spark represents calendar intervals losslessly as separate months, days, and microseconds. Comet needs the same representation across JVM/native boundaries and exact microsecond arithmetic in the native kernel.

What changes are included in this PR?

• Represent CalendarIntervalType as a Spark-tagged Arrow struct containing months: Int32, days: Int32, and microseconds: Int64.
• Preserve that logical type through Arrow conversion, protobuf serde, FFI, JVM readers/writers, codegen input/output, Scala UDF codegen, and native execution.
• Replace the datafusion-spark make_interval wrapper with an exact Decimal(18,6) microsecond kernel with Spark-compatible NULL and ANSI/TRY overflow behavior.
• Remove the obsolete incompatibility gate, ignored regressions, and unused datafusion-spark dependency.
• Port Spark 4.2.0 make_interval boundary and arity cases with source permalinks.
• Remove the obsolete duplicate codegen-dispatch benchmark case now that make_interval is fully native.

How are these changes tested?

• cargo test --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr preserves_spark_microsecond_range_and_overflow
• cargo check --manifest-path native/Cargo.toml -p datafusion-comet-spark-expr
• cargo check --manifest-path native/Cargo.toml -p datafusion-comet
• cargo fmt --manifest-path native/Cargo.toml --all -- --check
• make core
• Focused CometArrowStreamSuite CalendarInterval round-trip test
• Focused CometCodegenSuite CalendarInterval codegen test
• CometSqlFileTestSuite make_interval: 6/6 passed, 0 ignored
• CometSqlFileTestSuite calendar_interval: 1/1 passed
• Spotless, Scalastyle, and git diff --check

Benchmark

CometDatetimeExpressionBenchmark, 1,048,576 rows, Apple M4, JDK 17. The codegen-dispatch result is from parent commit 268849c0c; the full-native and Spark results are from this PR at 72997e9e8. Both Comet revisions used optimized native builds and the same query and input.

Execution path	Best time	Average time	Throughput	Per row	Relative to dispatch
Comet codegen dispatch	31 ms	33 ms	33.8 M rows/s	29.6 ns	1.00x
Comet full native	28 ms	30 ms	37.3 M rows/s	26.8 ns	1.11x
Spark	74 ms	81 ms	14.1 M rows/s	70.7 ns	0.42x

Full native is about 10% faster than codegen dispatch and 2.6x faster than Spark by best time.

[andygrove]

Thanks for the significant cleanup here — moving make_interval fully native and fixing the microsecond-precision loss is a real quality improvement, and the benchmark numbers show it. A few things worth addressing before merge.

ANSI overflow message diverges from Spark

Spark's MakeInterval.nullSafeEval catches the ArithmeticException from IntervalUtils.makeInterval and rethrows via arithmeticOverflowError(e.getMessage), where e.getMessage is Java's Math.addExact / multiplyExact string — literally "integer overflow" or "long overflow". Under ANSI, Spark reports:

[ARITHMETIC_OVERFLOW] integer overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

Comet's kernel always calls arithmetic_overflow_error(""interval"") at native/spark-expr/src/datetime_funcs/make_interval.rs:150, producing interval overflow instead. I confirmed this with a targeted expect_error(integer overflow) on make_interval(2147483647) — Spark's message matches, Comet's doesn't. The fixtures in this PR use expect_error(overflow. If necessary set), which matches both and hides the divergence.

Could the free make_interval function return the specific label (""integer"" for the year/month and week/day paths, ""long"" for the microsecond path) so arithmetic_overflow_error gets the same string Spark produces? A tighter expect_error in make_interval_ansi.sql would then pin the parity going forward.

Utils.toArrowType CalendarIntervalType branch is now unreachable

With the new case CalendarIntervalType => branch in toArrowField at spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala:212, the corresponding arm of toArrowType at line 172 is unreachable. If a future caller ever hits it directly, they get an unnamed, untagged ArrowType.Struct.INSTANCE with no children, and isCalendarIntervalStructField would fail to identify it on the round-trip. Would it be safer to make that arm throw with a message pointing callers at toArrowField, since the tagged struct can't be produced from an ArrowType alone?

The _dispatch SQL fixtures no longer exercise a distinct path

With MakeInterval unconditionally native, make_interval_dispatch.sql and make_interval_dispatch_ansi.sql run through the same kernel as make_interval.sql / make_interval_ansi.sql. The dispatch name is now misleading and the queries mostly duplicate coverage. Could these either be removed, or repurposed with -- Config: spark.comet.exec.enabled=false at the top so they cover the pure-Spark fallback path instead?

Argument downcasts .unwrap() where sibling kernels return errors

At native/spark-expr/src/datetime_funcs/make_interval.rs:109-118, the six Int32Array and one Decimal128Array downcasts unconditionally .unwrap(). This is unreachable today because the planner inserts a CastExpr for any input whose type differs from the Signature::exact, but SparkMakeDate in the same directory uses .ok_or_else(|| DataFusionError::Execution(...))? for the same pattern. Matching that style would give a diagnostic instead of a panic if a future serde change ever bypasses the cast.

Boundary test coverage: only positive extreme

The Int.MaxValue boundary row in make_interval.sql covers the positive side of the checked arithmetic. It might be worth adding a mirror row that pushes at least one of year/month/week/day toward Int.MinValue with a negative secs at the boundary, so the negative side of checked_add / checked_mul is validated against Spark too. Spark's own IntervalExpressionsSuite only tests the positive case, but with the native kernel the two sides are separate branches of i32::checked_mul and it's cheap to cover both.

End-to-end test for chained native consumption

calendar_interval.sql covers the shuffle case for array<interval>, but there's no test where a native make_interval result flows directly into another native step. If FFI (or any other stage on the interval column's path) ever drops the SPARK::calendarInterval::struct metadata on the months child, isCalendarIntervalStructField would silently degrade the type to an anonymous StructType(months, days, microseconds) and downstream CalendarInterval consumers would break with no clear error. A query like SELECT date '2020-01-01' + make_interval(1, 2, 3, 4, 5, 6, 7.123456) FROM t or a similar chained native consumer would pin the metadata-preservation invariant.

---

I ran CometSqlFileTestSuite make_interval locally on this branch — all 6 files pass. I also probed several extreme non-ANSI boundary cases (both signs of i32::MAX years/hours/minutes with -999999999999.999999 and +999999999999.999999 secs) and results match Spark. The change looks solid; the items above are all in-band for this PR.