Skip to content

chore: drop Spark 3.4 support #5358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
manuzhang wants to merge 2 commits into
base: main
Choose a base branch
from
Open

chore: drop Spark 3.4 support #5358

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4ed7982
chore: drop Spark 3.4 support
manuzhang Aug 14, 2026
5b0098a
chore: fix workflow README formatting
manuzhang Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions .claude/skills/audit-comet-expression/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: audit-comet-expression
description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.
description: Audit an existing Comet expression for correctness and test coverage. Studies the Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1, reviews the Comet and DataFusion implementations, identifies missing test coverage, and offers to implement additional tests.
argument-hint: <expression-name>
---

Expand Down Expand Up @@ -29,7 +29,7 @@ Audit the Comet implementation of the `$ARGUMENTS` expression for correctness an

This audit covers:

1. Spark implementation across versions 3.4.3, 3.5.8, 4.0.1, and 4.1.1
1. Spark implementation across versions 3.5.8, 4.0.1, and 4.1.1
2. Comet Scala serde implementation
3. Comet Rust / DataFusion implementation
4. Existing test coverage (Comet SQL Tests and Comet Scala Tests)
Expand All @@ -43,7 +43,7 @@ Clone specific Spark version tags (use shallow clones to avoid polluting the wor

```bash
set -eu -o pipefail
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
if [ ! -d "$dir" ]; then
git clone --depth 1 --branch "$tag" https://github.com/apache/spark.git "$dir"
Expand All @@ -56,7 +56,7 @@ done
Search the Catalyst SQL expressions source:

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql/catalyst/src/main/scala" -name "*.scala" | \
Expand All @@ -67,7 +67,7 @@ done
If the expression is not found in catalyst, also check core:

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql" -name "*.scala" | \
Expand All @@ -90,7 +90,6 @@ For each Spark version, read the expression file and note:

Produce a concise diff summary of what changed between:

- 3.4.3 → 3.5.8
- 3.5.8 → 4.0.1
- 4.0.1 → 4.1.1

Expand All @@ -107,7 +106,7 @@ Pay attention to:
## Step 2: Locate the Spark Tests

```bash
for tag in v3.4.3 v3.5.8 v4.0.1 v4.1.1; do
for tag in v3.5.8 v4.0.1 v4.1.1; do
dir="/tmp/spark-${tag}"
echo "=== $tag ==="
find "$dir/sql" -name "*.scala" -path "*/test/*" | \
Expand Down Expand Up @@ -732,7 +731,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`.
other pages.
- Add (or update) a `## <function_name>` section, keeping sections alphabetically ordered.
- Under that heading, add one bullet per Spark version checked, each including:
- Spark version (e.g. 3.4.3, 3.5.8, 4.0.1, 4.1.1)
- Spark version (e.g. 3.5.8, 4.0.1, 4.1.1)
- Today's date
- A brief note for any version-specific finding (behavioral difference, known
incompatibility); omit the note if nothing notable.
Expand All @@ -744,7 +743,7 @@ used for the expression in `docs/source/user-guide/latest/expressions.md`.
Present the audit as:

1. **Expression Summary** - Brief description of what `$ARGUMENTS` does, its input/output types, and null behavior
2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1
2. **Spark Version Differences** - Summary of any behavioral or API differences across Spark 3.5.8, 4.0.1, and 4.1.1
3. **Comet Implementation Notes** - Summary of how Comet implements this expression and any concerns
4. **Coverage Gap Analysis** - The gap table from Step 5, plus implementation gaps
5. **Recommendations** - Prioritized list from Step 6
Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/implement-comet-expression/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ make

### 4. Run the audit skill

Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.4.3, 3.5.8, and 4.0.1 and produces a prioritized list of gaps.
Once the initial implementation passes its smoke test, run the `audit-comet-expression` skill on `$ARGUMENTS`. It compares the implementation and tests against Spark 3.5.8, 4.0.1, and 4.1.1 and produces a prioritized list of gaps.

### 5. Implement audit-recommended tests and iterate

Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/wire-datafusion-function/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ make
cd native && cargo clippy --all-targets --workspace -- -D warnings
```

Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.4.3 / 3.5.8 / 4.0.1 and surface coverage gaps; iterate on tests.
Then run `audit-comet-expression` on `$ARGUMENTS` to compare against Spark 3.5.8 / 4.0.1 / 4.1.1 and surface coverage gaps; iterate on tests.

The user generally runs tests themselves. If asked for a smoke test:

Expand Down
19 changes: 9 additions & 10 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ is a `workflow_call` reusable invoked from the umbrella.
(PR+push) (PR+push) (PR+push)
| | |
v v v
spark_3_4 / spark_4_1 iceberg_1_8 / 1_9
spark_4_1 iceberg_1_8 / 1_9
(push or PR + label) (push only)

reusable workflows invoked via `uses:`:
Expand All @@ -60,7 +60,6 @@ is a `workflow_call` reusable invoked from the umbrella.
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
| `spark_3_5` | PR or push, paths matched | Spark 3.5 sources |
| `spark_4_0` | PR or push, paths matched | Spark 4.0 sources |
| `spark_3_4` | push, **or** PR with `run-spark-3.4-tests` label | Spark 3.4 sources |
| `spark_4_1` | push, **or** PR with `run-spark-4.1-tests` label | Spark 4.1 sources |
| `iceberg_1_10` | PR or push, paths matched | Iceberg sources |
| `iceberg_1_8` | push only | Iceberg sources |
Expand All @@ -86,14 +85,14 @@ umbrella doesn't watch, or operate independently of the rest of CI:

## Reusable workflows (called by `ci.yml`)

| File | Called from `ci.yml` job(s) |
| --------------------------------- | -------------------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1` |
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` |
| File | Called from `ci.yml` job(s) |
| --------------------------------- | -------------------------------------------- |
| `pr_build_linux.yml` | `pr_build_linux` |
| `pr_build_macos.yml` | `pr_build_macos` |
| `pr_benchmark_check.yml` | `pr_benchmark_check` |
| `docs.yaml` | `docs` |
| `spark_sql_test_reusable.yml` | `spark_3_5`, `spark_4_0`, `spark_4_1` |
| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10` |

## Modifying path filters

Expand Down
26 changes: 4 additions & 22 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ jobs:
if: >-
github.event_name != 'pull_request' ||
github.event.action != 'labeled' ||
github.event.label.name == 'run-spark-3.4-tests' ||
github.event.label.name == 'run-spark-4.0-tests' ||
github.event.label.name == 'run-spark-4.1-tests' ||
github.event.label.name == 'run-iceberg-tests'
Expand Down Expand Up @@ -110,7 +109,6 @@ jobs:
build_macos: ${{ steps.compute.outputs.build_macos }}
benchmark: ${{ steps.compute.outputs.benchmark }}
docs: ${{ steps.compute.outputs.docs }}
spark_3_4: ${{ steps.compute.outputs.spark_3_4 }}
spark_3_5: ${{ steps.compute.outputs.spark_3_5 }}
spark_4_0: ${{ steps.compute.outputs.spark_4_0 }}
spark_4_1: ${{ steps.compute.outputs.spark_4_1 }}
Expand All @@ -137,7 +135,7 @@ jobs:
run: |
set -euo pipefail
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do
for key in build_linux build_macos benchmark docs spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do
echo "${key}=true" >> "$GITHUB_OUTPUT"
done
exit 0
Expand Down Expand Up @@ -201,22 +199,6 @@ jobs:
(github.event_name == 'push' || github.event_name == 'workflow_dispatch')
uses: ./.github/workflows/docs.yaml

spark_3_4:
name: Spark SQL Tests (Spark 3.4)
needs: changes
# Main-only by default; PRs need the `run-spark-3.4-tests` label.
if: |
needs.changes.outputs.spark_3_4 == 'true' &&
(github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'run-spark-3.4-tests')))
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '3.4'
spark-full: '3.4.3'
java: 11

spark_3_5:
name: Spark SQL Tests (Spark 3.5)
needs: changes
Expand Down Expand Up @@ -277,9 +259,9 @@ jobs:
with:
iceberg-short: '1.8'
iceberg-full: '1.8.1'
spark-short: '3.4'
spark-full: '3.4.3'
java: 11
spark-short: '3.5'
spark-full: '3.5.9'
java: 17

iceberg_1_9:
name: Iceberg Spark SQL Tests (Iceberg 1.9)
Expand Down
9 changes: 1 addition & 8 deletions .github/workflows/pr_build_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,9 +83,6 @@ jobs:
strategy:
matrix:
profile:
- name: "Spark 3.4, JDK 11, Scala 2.12"
java_version: "11"
maven_opts: "-Pspark-3.4 -Pscala-2.12"
- name: "Spark 3.5, JDK 17, Scala 2.12"
java_version: "17"
maven_opts: "-Pspark-3.5 -Pscala-2.12"
Expand Down Expand Up @@ -276,10 +273,6 @@ jobs:
# the goal with these profiles is to get coverage of all Java, Scala, and Spark
# versions without testing all possible combinations, which would be overkill
profile:
- name: "Spark 3.4, JDK 11, Scala 2.12"
java_version: "11"
maven_opts: "-Pspark-3.4 -Pscala-2.12"

- name: "Spark 3.5, JDK 17, Scala 2.13"
java_version: "17"
maven_opts: "-Pspark-3.5 -Pscala-2.13"
Expand Down Expand Up @@ -336,7 +329,7 @@ jobs:
- name: "exec"
value: |
org.apache.comet.exec.CometAggregateSuite
org.apache.comet.exec.CometExec3_4PlusSuite
org.apache.comet.exec.CometExecCompatibilitySuite

@kazuyukitanimura kazuyukitanimura Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this used?

@manuzhang manuzhang Aug 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, CometExecCompatibilitySuite is renamed from CometExec3_4PlusSuite and used by all supported Spark versions.

org.apache.comet.exec.CometExecSuite
org.apache.comet.exec.CometGenerateExecSuite
org.apache.comet.exec.CometWindowExecSuite
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr_build_macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ jobs:
- name: "exec"
value: |
org.apache.comet.exec.CometAggregateSuite
org.apache.comet.exec.CometExec3_4PlusSuite
org.apache.comet.exec.CometExecCompatibilitySuite
org.apache.comet.exec.CometExecSuite
org.apache.comet.exec.CometGenerateExecSuite
org.apache.comet.exec.CometWindowExecSuite
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/pyarrow_udf_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ on:
- "spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala"
- "spark/src/main/scala/org/apache/spark/sql/comet/CometMapInBatchExec.scala"
- "spark/src/main/scala/org/apache/spark/sql/comet/shims/MapInBatchInfo.scala"
- "spark/src/main/spark-3.4/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
- "spark/src/main/spark-3.5/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
- "spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
- "spark/src/main/spark-4.1/org/apache/spark/sql/comet/shims/ShimCometMapInBatch.scala"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/spark_sql_writer_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
run: |
# Map each supported Spark minor version to its full version + JDK.
# Mirrors ci.yml's per-version reusable invocations (default-on PR
# versions only; 3.4 and 4.0 are label-gated and not offered here).
# versions only; 4.0 is label-gated and not offered here).
case "${{ inputs.spark-version }}" in
3.5) spark_full=3.5.9; java=17 ;;
4.1) spark_full=4.1.3; java=17 ;;
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ benefits of Comet's acceleration capabilities without disrupting your Spark appl

## Getting Started

Comet supports Apache Spark 3.4, 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the
Comet supports Apache Spark 3.5, 4.0, and 4.1, and provides experimental support for Spark 4.2. See the
[installation guide](https://datafusion.apache.org/comet/user-guide/installation.html) for the detailed
version, Java, and Scala compatibility matrix.

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ docker push localhost:32000/apache/datafusion-comet-tpcbench:latest
export SPARK_MASTER=k8s://https://127.0.0.1:16443
export COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest
# Location of Comet JAR within the Docker image
export COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.5.0-SNAPSHOT.jar
export COMET_JAR=/opt/spark/jars/comet-spark-spark3.5_2.12-0.5.0-SNAPSHOT.jar

$SPARK_HOME/bin/spark-submit \
--master $SPARK_MASTER \
Expand Down
25 changes: 0 additions & 25 deletions dev/ci/compute-changes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,36 +89,13 @@
"spark/src/main/scala/org/apache/comet/expressions/**",
"spark/src/main/spark-*/**",
],
"spark_3_4": [
"native/**/src/**",
"native/**/Cargo.toml",
"native/Cargo.lock",
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/spark-3.5/**",
"!spark/src/main/spark-4.0/**",
"!spark/src/main/spark-4.1/**",
"!spark/src/main/spark-4.2/**",
"!spark/src/main/spark-4.x/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
"spark/pom.xml",
"dev/diffs/3.4.3.diff",
"pom.xml",
"rust-toolchain.toml",
".github/workflows/ci.yml",
".github/workflows/spark_sql_test_reusable.yml",
".github/actions/setup-builder/**",
".github/actions/setup-spark-builder/**",
],
"spark_3_5": [
"native/**/src/**",
"native/**/Cargo.toml",
"native/Cargo.lock",
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/spark-3.4/**",
"!spark/src/main/spark-4.0/**",
"!spark/src/main/spark-4.1/**",
"!spark/src/main/spark-4.2/**",
Expand All @@ -140,7 +117,6 @@
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/spark-3.4/**",
"!spark/src/main/spark-3.5/**",
"!spark/src/main/spark-3.x/**",
"!spark/src/main/spark-4.1/**",
Expand All @@ -162,7 +138,6 @@
"common/src/main/**",
"common/pom.xml",
"spark/src/main/**",
"!spark/src/main/spark-3.4/**",
"!spark/src/main/spark-3.5/**",
"!spark/src/main/spark-3.x/**",
"!spark/src/main/spark-4.0/**",
Expand Down
Loading
Loading