Describe the bug
Under ANSI mode, abs on an integer minimum raises an overflow error whose type name comes from
Arrow rather than from Spark. Comet reports Int64 overflow where Spark reports long overflow.
native/spark-expr/src/math_funcs/abs.rs passes Arrow-style type names (Int8, Int16, Int32,
and Int64) as the overflow from_type, so the rendered message diverges from Spark's byte,
short, integer, and long. The names are hard-coded at each arm, four on the array path through
ansi_compute_op! and four on the scalar path.
This is user-visible: the message is what a Spark user sees in the exception, and it is what a test
asserting on the error class parameters would compare against.
Steps to reproduce
SET spark.sql.ansi.enabled=true;
SELECT abs(col) FROM (SELECT CAST(-9223372036854775808 AS BIGINT) AS col);
Spark raises [ARITHMETIC_OVERFLOW] long overflow.
Comet raises the same error class with Int64 overflow.
The same divergence applies to INT (integer vs Int32).
Expected behavior
The error carries Spark's type name, so long overflow and integer overflow.
Additional context
This is item 5 ("wrong type names" in Abs) from #5071. That issue is now closed; #5162 addressed
item 4, while this divergence remains on main. I asked on #5071 whether the remaining items should
reopen it or be tracked separately and did not want to leave it sitting, so this issue tracks the
Abs mismatch as a self-contained fix. Happy to move it back under #5071 if you would rather track
it there.
The mapping is not uniform across supported Spark versions, and that constrains the fix:
integer and long match Spark on 3.4, 3.5 and 4.x.
byte and short match 4.x only. On 3.4 and 3.5, Abs routes those two widths to
QueryExecutionErrors.unaryMinusCauseOverflowError, which raises _LEGACY_ERROR_TEMP_2043 with
- <sqlValue> caused overflow. rather than ARITHMETIC_OVERFLOW, so no single string satisfies
every version. 4.0 sends all four widths through MathUtils.negateExact.
I verified this against the 3.4.3, 3.5.8, 4.0.2, and 4.1.2 jars.
Worth noting that the existing abs_ansi.sql fixture cannot catch this: every assertion is
expect_error(overflow), which Int64 overflow satisfies just as happily as long overflow.
Decimal types are left alone here. Decimal128 and Decimal256 overflow goes through a different
Spark path and needs its own analysis.
I have a fix and tests ready and can send a PR.
Describe the bug
Under ANSI mode,
abson an integer minimum raises an overflow error whose type name comes fromArrow rather than from Spark. Comet reports
Int64 overflowwhere Spark reportslong overflow.native/spark-expr/src/math_funcs/abs.rspasses Arrow-style type names (Int8,Int16,Int32,and
Int64) as the overflowfrom_type, so the rendered message diverges from Spark'sbyte,short,integer, andlong. The names are hard-coded at each arm, four on the array path throughansi_compute_op!and four on the scalar path.This is user-visible: the message is what a Spark user sees in the exception, and it is what a test
asserting on the error class parameters would compare against.
Steps to reproduce
Spark raises
[ARITHMETIC_OVERFLOW] long overflow.Comet raises the same error class with
Int64 overflow.The same divergence applies to
INT(integervsInt32).Expected behavior
The error carries Spark's type name, so
long overflowandinteger overflow.Additional context
This is item 5 ("wrong type names" in
Abs) from #5071. That issue is now closed; #5162 addresseditem 4, while this divergence remains on
main. I asked on #5071 whether the remaining items shouldreopen it or be tracked separately and did not want to leave it sitting, so this issue tracks the
Absmismatch as a self-contained fix. Happy to move it back under #5071 if you would rather trackit there.
The mapping is not uniform across supported Spark versions, and that constrains the fix:
integerandlongmatch Spark on 3.4, 3.5 and 4.x.byteandshortmatch 4.x only. On 3.4 and 3.5,Absroutes those two widths toQueryExecutionErrors.unaryMinusCauseOverflowError, which raises_LEGACY_ERROR_TEMP_2043with- <sqlValue> caused overflow.rather thanARITHMETIC_OVERFLOW, so no single string satisfiesevery version. 4.0 sends all four widths through
MathUtils.negateExact.I verified this against the 3.4.3, 3.5.8, 4.0.2, and 4.1.2 jars.
Worth noting that the existing
abs_ansi.sqlfixture cannot catch this: every assertion isexpect_error(overflow), whichInt64 overflowsatisfies just as happily aslong overflow.Decimal types are left alone here.
Decimal128andDecimal256overflow goes through a differentSpark path and needs its own analysis.
I have a fix and tests ready and can send a PR.