[EPIC] Add support for Spark 4.0

[andygrove]

What is the problem the feature request solves?

Spark 4 will soon be available, so we should coordinate the work to add support. We already have partial support for Spark 4.0.0-preview1.

Let's use this issue to track the work and create issues as needed.

• Feat: Support Spark 4.0.0 part1 #1830
• Remove COMET_ANSI_MODE_ENABLED config #1974
• Update 4.0.0.diff to reflect recent improvements in 3.5.5.diff #1846
• Add support for StringDecode in Spark 4.0.0 #1942
• Add support for MapSort expression in Spark 4.0.0 #1941
• Fix listagg-collation.sql test in Spark 4.0.0 #1947
• Investigate failing tests with Spark 4.0.0 #1948
• Reinstate macOS CI builds of Comet with Spark 4.0.0 #1949
• Enable auto scan mode for Spark 4.0.0 #1967
• Fix Spark 4.0 sql tests #551
• [EPIC] Fully support ANSI mode #313
• [Spark 4.0.0] Can't zip RDDs with unequal numbers of partitions: List(9, 9, 9, 17) #2408
• Perform audit of all changes noted in https://spark.apache.org/docs/latest/sql-migration-guide.html#upgrading-from-spark-sql-35-to-40 and file issues as needed
• Update release scripts to publish Comet jars for Spark 4.0.0 #1989
• Add support for to_json compatibility with Spark 4.0.0. #3920
• CometSparkToColumnarExec missing from AQE plan on Spark 4 (two skipped tests) #4031

Describe the potential solution

No response

Additional context

No response

[dharanad]

ANSI Support EPIC #313

[kazuyukitanimura]

Investigate Guava duplicated classes #1968

[coderfender]

I can work on some of the low hanging issues @andygrove @kazuyukitanimura