build: Add spark-4.1 profile and shims

[manuzhang]

Which issue does this PR close?

First step of #2792.

Rationale for this change

What changes are included in this PR?

1. Add spark-4.1 profile with minor shim version spark-4.1
2. Move src/main/spark-4.0 to src/main/spark-4.x for common shim classes in spark-4.0 and spark-4.1.
3. Add CometSumShim and ShimSQLConf for spark-4.0 and spark-4.1 specific shims respectively
4. Add MapStatusBuilder.scala to access org.apache.spark.scheduler.MapStatus in java. MapStatus has added a constructor argument in Spark 4.1, and only kept compatibility for scala codes.

How are these changes tested?

Added UTs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.50%. Comparing base (f09f8af) to head (d3eed18).
⚠️ Report is 845 commits behind head on main.

Files with missing lines	Patch %	Lines
.../comet/execution/shuffle/CometShuffleManager.scala	25.00%	2 Missing and 1 partial ⚠️
...main/scala/org/apache/comet/serde/aggregates.scala	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2829      +/-   ##
============================================
+ Coverage     56.12%   59.50%   +3.38%     
- Complexity      976     1384     +408     
============================================
  Files           119      168      +49     
  Lines         11743    15561    +3818     
  Branches       2251     2582     +331     
============================================
+ Hits           6591     9260    +2669     
- Misses         4012     5002     +990     
- Partials       1140     1299     +159     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[manuzhang]

@andygrove @coderfender Please help review. Do we need to pass all tests now?

[andygrove]

Test failure:

CometExec3_4PlusSuite:
- subquery limit: limit with offset should return correct results (565 milliseconds)
- offset (602 milliseconds)
- test BloomFilterMightContain can take a constant value input (231 milliseconds)
- test NULL inputs for BloomFilterMightContain (251 milliseconds)
- test BloomFilterMightContain from random input *** FAILED *** (511 milliseconds)
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 41.0 failed 1 times, most recent failure: Lost task 1.0 in stage 41.0 (TID 102) (localhost executor driver): org.apache.comet.CometNativeException: assertion `left == right` failed: Unsupported BloomFilter version: 2, expecting version: 1
  left: 2
 right: 1
        at comet::errors::init::{{closure}}(__internal__:0)
        at std::panicking::panic_with_hook(__internal__:0)
        at std::panicking::panic_handler::{{closure}}(__internal__:0)
        at std::sys::backtrace::__rust_end_short_backtrace(__internal__:0)
        at __rustc::rust_begin_unwind(__internal__:0)
        at core::panicking::panic_fmt(__internal__:0)
        at core::panicking::assert_failed_inner(__internal__:0)
        at core::panicking::assert_failed(__internal__:0)
        at <datafusion_comet_spark_expr::bloom_filter::spark_bloom_filter::SparkBloomFilter as core::convert::From<&[u8]>>::from(__internal__:0)

[andygrove]

@andygrove @coderfender Please help review. Do we need to pass all tests now?

Yes, we either need tests to pass, or we can potentially disable some specific tests by updating the diff file, and file issues to resolve those failures.

[manuzhang]

@andygrove Sure, since Spark 4.1.1 is to be released soon, I will check again after upgrading to 4.1.1