deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest, MSRV to 1.88 #2729

mbutrovich · 2025-11-07T16:43:51Z

Which issue does this PR close?

Closes #2719.

Rationale for this change

What changes are included in this PR?

Bump DataFusion to 51 and adjust for API changes
Bump Arrow 57 and adjust for API changes
Bump Iceberg to latest now that it has DF 51 and Arrow 57
Bump Rust MSRV to 1.88
Refactor partition value serialization in CometIcebergNativeScan to consolidate redundant code

How are these changes tested?

Existing tests (including Iceberg suite), plus two new CometIcebergNativeSuite tests I added while debugging issues upgrading.

…date. Remove datafusion-sql dependency to improve build times.

mbutrovich · 2025-11-07T16:46:07Z

Marking as draft since this is just for testing until DataFusion 51.0.0 crates are available.

codecov-commenter · 2025-11-07T17:18:28Z

Codecov Report

❌ Patch coverage is 50.00000% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.35%. Comparing base (f09f8af) to head (de5ed17).
⚠️ Report is 753 commits behind head on main.

Files with missing lines	Patch %	Lines
.../comet/serde/operator/CometIcebergNativeScan.scala	47.54%	27 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2729      +/-   ##
============================================
+ Coverage     56.12%   59.35%   +3.23%     
- Complexity      976     1369     +393     
============================================
  Files           119      167      +48     
  Lines         11743    15334    +3591     
  Branches       2251     2545     +294     
============================================
+ Hits           6591     9102    +2511     
- Misses         4012     4945     +933     
- Partials       1140     1287     +147

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

# Conflicts: # native/Cargo.lock

mbutrovich · 2025-11-19T19:59:42Z

DF 51.0.0 crates are available, but we're currently planning to review and merge this after the Comet 0.12.0 release. Keeping this as a draft for now.

# Conflicts: # native/Cargo.lock # native/Cargo.toml

comphead · 2025-12-08T21:55:54Z

@mbutrovich are you planning to continue on this?

mbutrovich · 2025-12-09T15:19:45Z

@mbutrovich are you planning to continue on this?

Yep, we're blocked on apache/iceberg-rust#1899

# Conflicts: # native/Cargo.lock

mbutrovich · 2025-12-10T19:01:04Z

apache/iceberg-rust#1921 merged so I'm hoping CI will go green now and we can get this reviewed.

…ed columns (#1922) ## Which issue does this PR close? See #1824 (comment) and #1914 (comment) ## What changes are included in this PR? This restores the behavior in `record_batch_transformer.rs`'s `constants_map` function to pre-#1824 behavior where `NULL`s are not inserted into the constants map, and instead are just skipped. This allows the column projection rules for missing partition values to default to `NULL`. ## Are these changes tested? New test, and running the entire Iceberg Java suite via DataFusion Comet in apache/datafusion-comet#2729.

# Conflicts: # native/spark-expr/src/math_funcs/internal/make_decimal.rs

comphead · 2025-12-11T17:34:53Z

native/core/src/parquet/parquet_exec.rs

                file_source,
            )
-            .with_projection(Some(projection_vector))
+            .with_projection_indices(Some(projection_vector))


FYI It was an issue with Q13 projection pushdown with this method

apache/datafusion#19094 (review)

comphead · 2025-12-11T17:40:11Z

native/spark-expr/src/agg_funcs/avg_decimal.rs

+        Decimal128(precision, scale) => {
+            // In the spark, the result type is DECIMAL(min(38,precision+4), min(38,scale+4)).
+            // Ref: https://github.com/apache/spark/blob/fcf636d9eb8d645c24be3db2d599aba2d7e2955a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala#L66
+            let new_precision = DECIMAL128_MAX_PRECISION.min(*precision + 4);


comphead · 2025-12-11T17:44:11Z

spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala

+   */
+  private def partitionValueToJson(fieldTypeStr: String, value: Any): JValue = {
+    fieldTypeStr match {
+      case t if t.startsWith("timestamp") =>


any chance type name can be case sensitive?

From looking at the Iceberg code, doesn't look like it. They're all lowercase.

comphead · 2025-12-11T17:44:53Z

spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala

+                      // Schema selection logic:
+                      // 1. If hasDeletes=true: Use taskSchema (file-specific schema) because
+                      // delete files reference specific schema versions and we need exact schema
+                      // matching for MOR.


Suggested change

// matching for MOR.

// matching for merge-on-read.

comphead

Thanks @mbutrovich epic PR!

andygrove · 2025-12-12T01:30:24Z

I ran local benchmarks with scan modes auto and native_datafusion and did not see any change from main branch

Bump DF to latest main commit to test 51.0.0, bump Arrow-rs, cargo up…

6a8a3a4

…date. Remove datafusion-sql dependency to improve build times.

mbutrovich mentioned this pull request Nov 7, 2025

Release DataFusion 51.0.0 (Nov 2025) apache/datafusion#17558

Closed

37 tasks

mbutrovich marked this pull request as draft November 7, 2025 16:45

Fix "sql" feature dependency for native tests.

268e31b

mbutrovich added 2 commits November 7, 2025 14:44

Fix float normalize_nan bug?

57da62d

Remove unused imports.

eb95400

mbutrovich mentioned this pull request Nov 7, 2025

fix: NormalizeNaNAndZero::children() returns child's child #2732

Merged

mbutrovich changed the title ~~chore: upgrade to DataFusion 51.0.0 and Arrow-rs 57.0.0~~ deps: upgrade to DataFusion 51.0.0 and Arrow-rs 57.0.0 Nov 7, 2025

mbutrovich added 5 commits November 11, 2025 12:17

Switch to branch-51, run cargo update.

5399550

Bump df51.

c1d8a06

Merge branch 'main' into df51

d63f5cf

# Conflicts: # native/Cargo.lock

Merge branch 'main' into df51

ed0c8df

Move to crate of df51.

69bd15d

Merge branch 'main' into df51

736ffa9

# Conflicts: # native/Cargo.lock # native/Cargo.toml

mbutrovich mentioned this pull request Dec 4, 2025

deps: bump DataFusion to 51, Arrow to 57, pyo to 0.26 apache/iceberg-rust#1899

Merged

mbutrovich added 4 commits December 9, 2025 11:21

fix prost

8e10de0

Merge branch 'main' into df51

eaa488d

# Conflicts: # native/Cargo.lock

update

4bcba5e

Switch to DF51 iceberg-rust branch.

4cbbc3d

mbutrovich changed the title ~~deps: upgrade to DataFusion 51.0.0 and Arrow-rs 57.0.0~~ deps: [iceberg] upgrade to DataFusion 51.0.0 and Arrow-rs 57.0.0 Dec 9, 2025

mbutrovich changed the title ~~deps: [iceberg] upgrade to DataFusion 51.0.0 and Arrow-rs 57.0.0~~ deps: [iceberg] upgrade to DataFusion 51 and Arrow-rs 57 Dec 9, 2025

mbutrovich mentioned this pull request Dec 9, 2025

Add support for null values in Datum apache/iceberg-rust#1914

Open

Skip serializing NULL partition values.

4c6cbb1

mbutrovich changed the title ~~deps: [iceberg] upgrade to DataFusion 51 and Arrow-rs 57~~ deps: [iceberg] upgrade to DataFusion 51 and Arrow 57 Dec 9, 2025

fix schema selection

dead41a

mbutrovich added 7 commits December 9, 2025 17:48

Fix format.

95bc89c

Fix format.

304c169

switch back to main iceberg-rust now that df51 is in there

bbc7501

cargo update

05dc2bb

Fix decimal128 partition values.

b8576c1

Merge branch 'main' into df51

1b45c81

go back to iceberg-rust main branch

cabd0c1

Forgot to run cargo update.

9e0bedb

mbutrovich changed the title ~~deps: [iceberg] upgrade to DataFusion 51 and Arrow 57~~ deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest Dec 10, 2025

test null identity-partitioned fix

d6b493a

mbutrovich mentioned this pull request Dec 10, 2025

fix: restore no-op logic in constants_map for NULL identity-partitioned columns apache/iceberg-rust#1922

Merged

back to main iceberg-rust

961957f

mbutrovich marked this pull request as ready for review December 11, 2025 12:35

Merge branch 'main' into df51

de5ed17

# Conflicts: # native/spark-expr/src/math_funcs/internal/make_decimal.rs

mbutrovich added bug Something isn't working dependencies Pull requests that update a dependency file and removed bug Something isn't working labels Dec 11, 2025

mbutrovich requested review from andygrove and comphead December 11, 2025 13:00

mbutrovich changed the title ~~deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest~~ deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest, MSRV to 1.88 Dec 11, 2025

comphead reviewed Dec 11, 2025

View reviewed changes

comphead approved these changes Dec 11, 2025

View reviewed changes

mbutrovich merged commit 97767c2 into apache:main Dec 11, 2025
172 of 173 checks passed

mbutrovich deleted the df51 branch December 12, 2025 15:00

deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest, MSRV to 1.88 #2729

deps: [iceberg] upgrade DataFusion to 51, Arrow to 57, Iceberg to latest, MSRV to 1.88 #2729

Uh oh!

Conversation

mbutrovich commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

mbutrovich commented Nov 7, 2025

Uh oh!

codecov-commenter commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mbutrovich commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Dec 8, 2025

Uh oh!

mbutrovich commented Dec 9, 2025

Uh oh!

mbutrovich commented Dec 10, 2025

Uh oh!

comphead Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

comphead Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

mbutrovich Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

comphead Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andygrove commented Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mbutrovich commented Nov 7, 2025 •

edited

Loading

codecov-commenter commented Nov 7, 2025 •

edited

Loading

mbutrovich commented Nov 19, 2025 •

edited

Loading

comphead Dec 11, 2025 •

edited

Loading