[SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown

[goutamadwant]

What changes were proposed in this pull request?

Run SchemaPruning after PushVariantIntoScan in the early scan pushdown batch. This removes variant reconstruction projections that become unnecessary during variant pushdown and allows the Once batch to reach the same plan on its first application.

Add a Parquet V1 regression test covering a query that reads _metadata.file_path while a VARIANT column is referenced below a nondeterministic filter.

Closes #57659.

Why are the changes needed?

PushVariantIntoScan can make a variant reconstruction projection unnecessary after the earlier schema-pruning passes have completed. Reapplying the optimizer batch then removes that projection, which violates the idempotence requirement for a Once batch and causes the reported query to fail during optimization.

Running SchemaPruning once after variant pushdown makes the first application reach the stable plan.

Does this PR introduce any user-facing change?

Yes. Queries that select file metadata while pruning a VARIANT column below a nondeterministic filter no longer fail during optimization with a Once batch idempotence error.

How was this patch tested?

The new regression test was confirmed to fail before the optimizer change and pass afterward.

The following tests and checks passed:

• ./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite -- -z "SPARK-57659"'
• ./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanVectorizedSuite' (349 tests)
• ./build/sbt 'sql/scalastyle' 'sql/Test/scalastyle'
• git diff --check

The full ./dev/run-tests suite was not run locally.

Was this patch authored or co-authored using generative AI tooling?

AI was used to review the code and understand the existing codebase.

This contribution is my original work, and I license it under the project's open source license.

[felipepessoto]

@uros-b , @goutamadwant any idea why CI is not running?

[uros-b]

@goutamadwant @felipepessoto Please try to follow the instructions in https://github.com/apache/spark/pull/57809/checks?check_run_id=92506854083.

[uros-b]

For more information, please see section "Pull request" in https://spark.apache.org/contributing.html