[SPARK-55996][CORE] Remove default jdk.reflect.useDirectMethodHandle=false

[pan3793]

What changes were proposed in this pull request?

This PR removes the auto-added Java options -Djdk.reflect.useDirectMethodHandle=false, which was added via SPARK-40729 to workaround Spark Shell issues (caused by ClosureCleaner) on JDK 18 to 21.

Why are the changes needed?

-Djdk.reflect.useDirectMethodHandle=false is a backdoor to disable JEP 416: Reimplement Core Reflection with Method Handles optimization to use the legacy JDK behaviors on JDK 18 ~ 21. Now, the Spark Shell issue was resolved by SPARK-53226/SPARK-52088 (#52956), we can remove this workaround.

Does this PR introduce any user-facing change?

No, but if the users hit bugs with the ClonedIndyLambda mode on Java 21, they are still able to add -Djdk.reflect.useDirectMethodHandle=false via spark.driver.extraJavaOptions to switch back to the legacy approach.

How was this patch tested?

Pass GHA, also update the JsonBenchmark-jdk21-results.txt(Jackson was mentioned in JEP 416 benchmark cases).

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

will revert later

[pan3793]

Java 21 GHA passed at https://github.com/pan3793/spark/actions/runs/23118461195

reverted

[LuciferYang]

We'd better conduct a more thorough evaluation of the impact on performance, as this could lead to MethodHandle being used by default across the board to replace the traditional reflection-based implementation approach.

Personally, I would prefer to remove it only when Java 25 becomes the default JDK, as by that time it will no longer have any function or impact.

[pan3793]

@LuciferYang, I'm optimistic about this because the JEP 416 says:

This degradation may, however, not have much effect on the performance of real-world applications. We ran several serialization and deserialization benchmarks using real-world libraries and found no degradation ...

but I'm okay to hold this until Spark moves to JDK 25 as the default JDK, if you have concerns.

[sarutak]

I don’t think it’s necessary to remove it right now, but, if there are any issues, we can find them in the next preview release.
The change itself LGTM.