Skip to content

feat: Cast numeric (non int) to timestamp#3559

Merged
parthchandra merged 11 commits intoapache:mainfrom
coderfender:cast_float_to_timestamp
Mar 6, 2026
Merged

feat: Cast numeric (non int) to timestamp#3559
parthchandra merged 11 commits intoapache:mainfrom
coderfender:cast_float_to_timestamp

Conversation

@coderfender
Copy link
Contributor

@coderfender coderfender commented Feb 20, 2026

Which issue does this PR close?

Closes ##3560 .

Rationale for this change

Adding support for further more native cast support.

  1. Float / Double cast to Timestamp (with ANSI support)
  2. DecimalType -> Timestamp cast (with ANSI support)
  3. Boolean -> Timestamp cast

This should more or less close out the cast matrix in terms of support (barring other not planned casts such as Timestamp -> Int etc)

What changes are included in this PR?

How are these changes tested?

  1. Enabled unit tests in CometCastSuite.scala and added new ones (along with benchmarking scripts) on the rust side to test cast at all Eval Modes

@coderfender coderfender force-pushed the cast_float_to_timestamp branch from 41393a5 to 5aa49b0 Compare February 20, 2026 21:09
@coderfender coderfender changed the title feat: Cast non numeric to timestamp feat: Cast numeric (non ints) to timestamp Feb 20, 2026
@coderfender coderfender changed the title feat: Cast numeric (non ints) to timestamp feat: Cast numeric (non int) to timestamp Feb 20, 2026
@coderfender coderfender force-pushed the cast_float_to_timestamp branch from 5aa49b0 to 88e0c83 Compare February 21, 2026 00:42
@coderfender coderfender force-pushed the cast_float_to_timestamp branch from bd49514 to 4d3a6a3 Compare February 24, 2026 20:37
@coderfender
Copy link
Contributor Author

@andygrove I moved the 'assertDataFrameEquals' inside cast test to leverage existing Try and ANSI checks . This should help us gain confidence with native code and perhaps merge it

@andygrove
Copy link
Member

Test failure:

- cast FloatType to TimestampType *** FAILED *** (323 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 2197.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2197.0 (TID 2396) (localhost executor driver): org.apache.comet.CometNativeException: [CAST_INVALID_INPUT] The value 'NaN' of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

@coderfender
Copy link
Contributor Author

Thank you @andygrove There seems to be failure per spark version after I fixed the error message handling. I am making changes to fix tests with the right error message

@coderfender coderfender force-pushed the cast_float_to_timestamp branch 2 times, most recently from 67d6e37 to 0546da6 Compare February 27, 2026 20:36
@coderfender coderfender force-pushed the cast_float_to_timestamp branch from 0546da6 to 3f3dd19 Compare February 27, 2026 21:35
@coderfender
Copy link
Contributor Author

it seems like there is some regression between Spark's cast from Float / Double to Timestamp in terms handling extreme values . Investigating into this

@coderfender
Copy link
Contributor Author

Only spark 4 test is failing and it seems like we might not be using right exception parsing

@parthchandra
Copy link
Contributor

Only spark 4 test is failing and it seems like we might not be using right exception parsing

Would you like to wait forhttps://github.com//pull/3580 ?

@coderfender
Copy link
Contributor Author

@parthchandra sure !

@coderfender coderfender marked this pull request as draft March 3, 2026 06:24
@coderfender
Copy link
Contributor Author

@andygrove , marking it as a draft to not accidentally merge until @parthchandra 's PR is reviewed

@parthchandra parthchandra marked this pull request as ready for review March 3, 2026 17:25
@parthchandra
Copy link
Contributor

@coderfender I've marked this as ready for review if you could rebase and resolve the ci failures.

@coderfender
Copy link
Contributor Author

Sure thank you

@coderfender
Copy link
Contributor Author

It seems like the errors are transient @parthchandra . Can you please rerun failed actions whenever you get a chance

@coderfender coderfender force-pushed the cast_float_to_timestamp branch from 7bbe49b to e593089 Compare March 5, 2026 22:50
@coderfender
Copy link
Contributor Author

coderfender commented Mar 6, 2026

@parthchandra @andygrove , Rebased with main . Please review whenever you get a chance . Thank you

Copy link
Contributor

@parthchandra parthchandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. (pending ci)

@coderfender
Copy link
Contributor Author

Thank you

@parthchandra parthchandra merged commit 34bbe78 into apache:main Mar 6, 2026
159 of 160 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants