Skip to content

fix: Update TPC-DS q36a golden file for Spark 4.0 decimal UNION widening change#3915

Open
parthchandra wants to merge 4 commits intoapache:mainfrom
parthchandra:decimal-div-tpcds
Open

fix: Update TPC-DS q36a golden file for Spark 4.0 decimal UNION widening change#3915
parthchandra wants to merge 4 commits intoapache:mainfrom
parthchandra:decimal-div-tpcds

Conversation

@parthchandra
Copy link
Copy Markdown
Contributor

@parthchandra parthchandra commented Apr 8, 2026

Which issue does this PR close?

Running TPC-DS resulted in a schema mismatch error in q36a-v2.7

Rationale for this change

The golden file for TPC-DS v2.7 query q36a was originally generated against Spark 3.4, where DecimalPrecisionTypeCoercion.bounded() simply clamped precision/scale to the maximum (min(p, 38), min(s, 38)). This caused the UNION of decimal(37,20) and decimal(38,11) (the two branches of results_rollup) to widen to decimal(38,20).

SPARK-45905 changed the decimal least-common-type algorithm to prefer retaining integral digits over fractional digits when precision exceeds 38. The same UNION now correctly widens to decimal(38,11). Apache Spark's own TPCDSQueryTestSuite golden file reflects this.

What changes are included in this PR?

This adds a version specific golden file against Spark 4 to match the updated behavior. Also, decimal tests now also check that the schema matches with the one output by Spark.

How are these changes tested?

Running TPC-DS queries in CometTPCDSQueryBenchmark. Also unit tests.

@parthchandra parthchandra requested a review from andygrove April 8, 2026 20:41
Copy link
Copy Markdown
Contributor

@mbutrovich mbutrovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed PR description. This sounds like it was a journey to track down. Thanks @parthchandra!

@parthchandra
Copy link
Copy Markdown
Contributor Author

Oh. Now the schema fails for pre-4.0 versions of Spark. So now we need a version specific set of golden files!

@parthchandra
Copy link
Copy Markdown
Contributor Author

@mbutrovich thank you for the approval, but if you don't mind taking another look. I changed the implementation so that there is a version specific result file. Since only the one query is affected, there is only one additional file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants