Skip to content

ANSI abs integer overflow errors use Arrow-style type names instead of Spark's #5356

Description

@Smallfu666

Describe the bug

Under ANSI mode, abs on an integer minimum raises an overflow error whose type name comes from
Arrow rather than from Spark. Comet reports Int64 overflow where Spark reports long overflow.

native/spark-expr/src/math_funcs/abs.rs passes Arrow-style type names (Int8, Int16, Int32,
and Int64) as the overflow from_type, so the rendered message diverges from Spark's byte,
short, integer, and long. The names are hard-coded at each arm, four on the array path through
ansi_compute_op! and four on the scalar path.

This is user-visible: the message is what a Spark user sees in the exception, and it is what a test
asserting on the error class parameters would compare against.

Steps to reproduce

SET spark.sql.ansi.enabled=true;
SELECT abs(col) FROM (SELECT CAST(-9223372036854775808 AS BIGINT) AS col);

Spark raises [ARITHMETIC_OVERFLOW] long overflow.
Comet raises the same error class with Int64 overflow.

The same divergence applies to INT (integer vs Int32).

Expected behavior

The error carries Spark's type name, so long overflow and integer overflow.

Additional context

This is item 5 ("wrong type names" in Abs) from #5071. That issue is now closed; #5162 addressed
item 4, while this divergence remains on main. I asked on #5071 whether the remaining items should
reopen it or be tracked separately and did not want to leave it sitting, so this issue tracks the
Abs mismatch as a self-contained fix. Happy to move it back under #5071 if you would rather track
it there.

The mapping is not uniform across supported Spark versions, and that constrains the fix:

  • integer and long match Spark on 3.4, 3.5 and 4.x.
  • byte and short match 4.x only. On 3.4 and 3.5, Abs routes those two widths to
    QueryExecutionErrors.unaryMinusCauseOverflowError, which raises _LEGACY_ERROR_TEMP_2043 with
    - <sqlValue> caused overflow. rather than ARITHMETIC_OVERFLOW, so no single string satisfies
    every version. 4.0 sends all four widths through MathUtils.negateExact.

I verified this against the 3.4.3, 3.5.8, 4.0.2, and 4.1.2 jars.

Worth noting that the existing abs_ansi.sql fixture cannot catch this: every assertion is
expect_error(overflow), which Int64 overflow satisfies just as happily as long overflow.

Decimal types are left alone here. Decimal128 and Decimal256 overflow goes through a different
Spark path and needs its own analysis.

I have a fix and tests ready and can send a PR.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions