Skip to content

[SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table references in DML#57508

Open
anuragmantri wants to merge 1 commit into
apache:masterfrom
anuragmantri:self-ref-options
Open

[SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table references in DML#57508
anuragmantri wants to merge 1 commit into
apache:masterfrom
anuragmantri:self-ref-options

Conversation

@anuragmantri

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR fixes dynamic table options being silently dropped when the same table is referenced more than once in a single DML statement. Options are now re-applied on a per-query relation cache hit, so each reference keeps its own options.

Why are the changes needed?

The per-query relation cache in RelationResolution is keyed by catalog, namespace, name, and time travel spec, but not by options. When the same table is referenced twice in one statement with different options, the second reference gets a cache hit and reuses the first reference's options, so the options the user wrote on the second reference are silently ignored. This affects INSERT, UPDATE, and MERGE.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added end-to-end tests to the following suites, each asserting that both references keep their own options:

  • DataSourceV2OptionSuite (INSERT selecting from the same table)
  • MergeIntoTableSuiteBase
  • UpdateTableSuiteBase

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

@anuragmantri

Copy link
Copy Markdown
Contributor Author

@dongjoon-hyun @peter-toth - Please provide your review on this.

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @anuragmantri .

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-58330] [SQL] Prevent silent dropping of dynamic options on same table references in DML [SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table references in DML Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants