[SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` by MaxGekk · Pull Request #56094 · apache/spark

MaxGekk · 2026-05-24T20:21:16Z

What changes were proposed in this pull request?

This PR completes SPARK-56969 on top of the parser gating added in SPARK-56965 / #56041.

Extend TypeUtils.failUnsupportedDataType to recursively reject TimestampNTZNanosType and TimestampLTZNanosType when spark.sql.timestampNanosTypes.enabled is off.
Add UNSUPPORTED_TIMESTAMP_NANOS_TYPE with a message naming the conf key.
Expand the SQLConf doc for spark.sql.timestampNanosTypes.enabled and align its default with Utils.isTesting (mirroring spark.sql.timeType.enabled).
Add a short enablement note for preview nanos timestamp types in docs/sql-ref-datatypes.md.

Why are the changes needed?

Parser and JSON entry points already gate parameterized nanos timestamp types behind spark.sql.timestampNanosTypes.enabled, but analyzed schemas/plans could still surface these types through other paths (for example CREATE TABLE, connectors, or programmatic schemas) before downstream execution support is ready.

Analysis-time gating closes that gap and keeps behavior consistent with the existing preview flag.

Does this PR introduce any user-facing change?

Yes.

When spark.sql.timestampNanosTypes.enabled is off, analyzed schemas/plans containing TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) with p in [7, 9] now fail with UNSUPPORTED_TIMESTAMP_NANOS_TYPE.
The conf default is now Utils.isTesting instead of always false, so tests enable the preview by default while production remains off.
docs/sql-ref-datatypes.md documents how to enable the preview feature.

Unparameterized TIMESTAMP, TIMESTAMP_NTZ, and TIMESTAMP_LTZ behavior is unchanged.

Example:

SET spark.sql.timestampNanosTypes.enabled=false;
CREATE TABLE t (c TIMESTAMP_NTZ(9));
-- UNSUPPORTED_TIMESTAMP_NANOS_TYPE: ... Set "spark.sql.timestampNanosTypes.enabled" to "true" ...

How was this patch tested?

Added/updated unit tests:

TypeUtilsSuite: default conf behavior, analysis gating on/off, microsecond types unaffected
DataTypeParserSuite: explicitly disable conf when testing parser rejection
DataTypeSuite: explicitly disable conf when testing JSON rejection

Existing nanos parser/JSON tests continue to pass with the conf enabled.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor Auto

…cument preview conf Extend TypeUtils.failUnsupportedDataType to reject TimestampNTZNanosType and TimestampLTZNanosType when spark.sql.timestampNanosTypes.enabled is off, add UNSUPPORTED_TIMESTAMP_NANOS_TYPE, align the conf default with Utils.isTesting, and add a short enablement note in sql-ref-datatypes.md.

dongjoon-hyun

Do we still need [WIP] in the PR title, @MaxGekk ?

MaxGekk · 2026-05-25T06:18:48Z

@dongjoon-hyun The PR is ready for review. I removed the [WIP] tag.

MaxGekk · 2026-05-25T07:02:36Z

+        "types. Enabling this flag does not guarantee full SQL support: casts, Parquet read, " +
+        "typed literals, and other operations may still fail until their respective features " +
+        "are implemented.")
      .version("4.2.0")


@HyukjinKwon @cloud-fan @dongjoon-hyun I merged this config to master, will it be released in 4.2.0? Should I merge new features to branch-4.x?

Mirror the SQLConf default (Utils.isTesting) in DefaultSqlApiConf so no-session-bound paths (Spark Connect parsing without a session) match session-bound behavior. Drop the link to "Runtime SQL Configuration" in sql-ref-datatypes.md since the conf is .internal() and is not listed there. Co-authored-by: Isaac

MaxGekk · 2026-05-25T07:58:34Z

@stevomitric Please, have a look at the PR. It is related to your changes in #56041

stevomitric

Latest commit says "Co-authored-by: Isaac" but the description states "Generated-by: Cursor Auto".

Maybe squash the latest commits?

stevomitric · 2026-05-25T09:01:11Z

  }

+  def unsupportedTimestampNanosTypeError(): Throwable = {
+    new AnalysisException(


Why not reuse FEATURE_NOT_ENABLED ?

It makes sense. Reused.

stevomitric

LGTM.

Drop the dedicated UNSUPPORTED_TIMESTAMP_NANOS_TYPE error condition and route the analysis-time gate through the same FEATURE_NOT_ENABLED throw used by the parser path. Extract the throw into DataTypeErrors.timestampNanosTypesNotEnabledError so both gates share one source of truth. Short-circuit on the conf flag before traversing the data type recursively. Co-authored-by: Isaac

dongjoon-hyun

+1, LGTM.

MaxGekk · 2026-05-26T13:07:44Z

Waiting for @stevomitric PR #56112 be merged as soon as CI passed.

MaxGekk · 2026-05-26T15:47:15Z

Merging to master/4.x. Thank you, @stevomitric @dongjoon-hyun for review.

…pes.enabled` ### What changes were proposed in this pull request? This PR completes [SPARK-56969](https://issues.apache.org/jira/browse/SPARK-56969) on top of the parser gating added in [SPARK-56965](https://issues.apache.org/jira/browse/SPARK-56965) / [#56041](#56041). - Extend `TypeUtils.failUnsupportedDataType` to recursively reject `TimestampNTZNanosType` and `TimestampLTZNanosType` when `spark.sql.timestampNanosTypes.enabled` is off. - Add `UNSUPPORTED_TIMESTAMP_NANOS_TYPE` with a message naming the conf key. - Expand the SQLConf doc for `spark.sql.timestampNanosTypes.enabled` and align its default with `Utils.isTesting` (mirroring `spark.sql.timeType.enabled`). - Add a short enablement note for preview nanos timestamp types in `docs/sql-ref-datatypes.md`. Part of SPIP [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822). ### Why are the changes needed? Parser and JSON entry points already gate parameterized nanos timestamp types behind `spark.sql.timestampNanosTypes.enabled`, but analyzed schemas/plans could still surface these types through other paths (for example `CREATE TABLE`, connectors, or programmatic schemas) before downstream execution support is ready. Analysis-time gating closes that gap and keeps behavior consistent with the existing preview flag. ### Does this PR introduce _any_ user-facing change? Yes. - When `spark.sql.timestampNanosTypes.enabled` is off, analyzed schemas/plans containing `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` with `p` in `[7, 9]` now fail with `UNSUPPORTED_TIMESTAMP_NANOS_TYPE`. - The conf default is now `Utils.isTesting` instead of always `false`, so tests enable the preview by default while production remains off. - `docs/sql-ref-datatypes.md` documents how to enable the preview feature. Unparameterized `TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` behavior is unchanged. Example: ```sql SET spark.sql.timestampNanosTypes.enabled=false; CREATE TABLE t (c TIMESTAMP_NTZ(9)); -- UNSUPPORTED_TIMESTAMP_NANOS_TYPE: ... Set "spark.sql.timestampNanosTypes.enabled" to "true" ... ``` ### How was this patch tested? Added/updated unit tests: - TypeUtilsSuite: default conf behavior, analysis gating on/off, microsecond types unaffected - DataTypeParserSuite: explicitly disable conf when testing parser rejection - DataTypeSuite: explicitly disable conf when testing JSON rejection Existing nanos parser/JSON tests continue to pass with the conf enabled. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Auto Closes #56094 from MaxGekk/nanos-conf. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com> (cherry picked from commit 5477eea) Signed-off-by: Max Gekk <max.gekk@gmail.com>

MaxGekk added 2 commits May 24, 2026 22:12

Fix error-conditions.json

5502ebc

dongjoon-hyun reviewed May 25, 2026

View reviewed changes

MaxGekk changed the title ~~[WIP][SPARK-56969][SQL] Enhance the SQL config spark.sql.timestampNanosTypes.enabled~~ [SPARK-56969][SQL] Enhance the SQL config spark.sql.timestampNanosTypes.enabled May 25, 2026

MaxGekk commented May 25, 2026

View reviewed changes

stevomitric reviewed May 25, 2026

View reviewed changes

stevomitric approved these changes May 25, 2026

View reviewed changes

dongjoon-hyun approved these changes May 25, 2026

View reviewed changes

Bump version: 4.2.0 -> 4.3.0

58a6355

MaxGekk closed this in 5477eea May 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled`#56094

[SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled`#56094
MaxGekk wants to merge 5 commits into
apache:masterfrom
MaxGekk:nanos-conf

MaxGekk commented May 24, 2026

Uh oh!

dongjoon-hyun left a comment

Uh oh!

MaxGekk commented May 25, 2026

Uh oh!

MaxGekk May 25, 2026

Uh oh!

MaxGekk commented May 25, 2026

Uh oh!

stevomitric left a comment

Uh oh!

stevomitric May 25, 2026

Uh oh!

MaxGekk May 25, 2026

Uh oh!

stevomitric left a comment

Uh oh!

dongjoon-hyun left a comment

Uh oh!

MaxGekk commented May 26, 2026

Uh oh!

MaxGekk commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MaxGekk commented May 24, 2026

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented May 25, 2026

Uh oh!

MaxGekk May 25, 2026

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented May 25, 2026

Uh oh!

stevomitric left a comment

Choose a reason for hiding this comment

Uh oh!

stevomitric May 25, 2026

Choose a reason for hiding this comment

Uh oh!

MaxGekk May 25, 2026

Choose a reason for hiding this comment

Uh oh!

stevomitric left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented May 26, 2026

Uh oh!

MaxGekk commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants