[AMORO-4044] Fix partition mapping bug in TableEntriesScan for tables with evolved PartitionSpec by j1wonpark · Pull Request #4085 · apache/amoro

j1wonpark · 2026-02-13T11:32:09Z

Why are the changes needed?

The entries metadata table returns a unified super-struct partition containing fields from all PartitionSpecs. In TableEntriesScan, both buildDataFile() and buildDeleteFile() passed this unified partition directly to withPartition(), which causes a partition struct mismatch for files written under a non-latest PartitionSpec. This is the root cause of #4044.

This PR fixes the issue at the TableEntriesScan level by projecting the unified partition to the spec-specific partition type using StructProjection.createAllowMissing(), so that all callers of TableEntriesScan get correct partition values regardless of spec evolution.

Close #4044.

Brief change log

Add projectPartition() method that uses StructProjection.createAllowMissing() to project the unified partition struct to the file's spec-specific partition type
Apply the projection in both buildDataFile() and buildDeleteFile()
Cache the unified partition type via Partitioning.partitionType(table) for reuse

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
no
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
not applicable

j1wonpark · 2026-02-15T12:19:23Z

Hi @majin1102 @zhoujinsong, could you please review this PR? This fix addresses a partition mapping bug in TableEntriesScan for Iceberg tables with evolved PartitionSpec (AMORO-4044). Your expertise in this area would be greatly appreciated. Thanks!

… with evolved PartitionSpec The entries metadata table returns a unified super-struct partition containing fields from all PartitionSpecs. buildDataFile() and buildDeleteFile() passed this directly to withPartition(), causing partition mismatch after spec evolution. Fix by projecting the unified partition to the spec-specific type using StructProjection. Signed-off-by: Jiwon Park <jpark92@outlook.kr>

lintingbin · 2026-02-27T03:26:40Z

We encountered the same issue internally and have tested this branch — all our internal tests passed. LGTM from the testing perspective.

One minor pre-existing issue I noticed (not introduced by this PR): the LOG.info calls in buildDataFile() and buildDeleteFile() use %S (printf-style) instead of {} (SLF4J-style) for the filePath parameter, so the file path is never actually substituted into the log message. I'll submit a separate PR to fix that.

j1wonpark · 2026-03-05T11:56:54Z

Hi @lintingbin, would you mind reviewing this PR?

It addresses the partition mismatch bug reported in #4044 — specifically fixing TableEntriesScan to correctly project the unified partition struct to each file's spec-specific partition type using StructProjection.createAllowMissing().

Thanks!

lintingbin · 2026-03-05T12:06:25Z

#4047 This PR is the fix proposed by our colleague, but I feel your solution is a better approach. We've already tested it internally on your branch, and everything looks good. I think it's ready to merge. @xxubai @zhoujinsong Please take a look at this.

xxubai · 2026-03-05T12:42:12Z

amoro-format-iceberg/src/main/java/org/apache/amoro/scan/TableEntriesScan.java

    return lazyIndexOfDataFileType.get(fieldName);
  }

+  private StructLike projectPartition(PartitionSpec spec, StructLike partition) {


It seems this function duplicates the logic in PartitionUtil.coercePartition(...). Maybe we can reuse that method directly.

Thanks for the suggestion! At first glance they do look similar, but the projection direction is actually reversed.

coercePartition projects spec-specific → unified (input: spec.partitionType(), output: partitionType):

StructProjection.createAllowMissing(spec.partitionType(), partitionType);

projectPartition here projects unified → spec-specific (input: unifiedPartitionType(), output: spec.partitionType()):

StructProjection.createAllowMissing(unifiedPartitionType(), spec.partitionType());

In TableEntriesScan, the partition data is read from the entries metadata table, which uses a unified partition schema (Partitioning.partitionType(table)). But DataFiles.builder(spec).withPartition() expects the spec-specific layout. So we need the unified → spec direction, which is the opposite of what coercePartition does.

xxubai · 2026-03-05T12:43:26Z

amoro-format-iceberg/src/main/java/org/apache/amoro/scan/TableEntriesScan.java

      StructLike partition =
          fileRecord.get(dataFileFieldIndex(DataFile.PARTITION_NAME), StructLike.class);
-      builder.withPartition(partition);
+      builder.withPartition(projectPartition(spec, partition));


Suggested change

builder.withPartition(projectPartition(spec, partition));

builder.withPartition(PartitionUtil.coercePartition(unifiedPartitionType(), spec, partition));

xxubai · 2026-03-05T12:43:45Z

amoro-format-iceberg/src/main/java/org/apache/amoro/scan/TableEntriesScan.java

      StructLike partition =
          fileRecord.get(dataFileFieldIndex(DataFile.PARTITION_NAME), StructLike.class);
-      builder.withPartition(partition);
+      builder.withPartition(projectPartition(spec, partition));


Suggested change

builder.withPartition(projectPartition(spec, partition));

builder.withPartition(PartitionUtil.coercePartition(unifiedPartitionType(), spec, partition));

xxubai · 2026-03-06T02:30:46Z

amoro-format-iceberg/src/main/java/org/apache/amoro/scan/TableEntriesScan.java

    return lazyIndexOfDataFileType.get(fieldName);
  }

+  private StructLike projectPartition(PartitionSpec spec, StructLike partition) {


j1wonpark mentioned this pull request Feb 14, 2026

[AMORO-4044] Return correct partition for delete files in iceberg tables whose partition spec have changed #4047

Closed

3 tasks

j1wonpark force-pushed the fix/AMORO-4044-partition-mapping-bug branch from 61c7c21 to 7c4f02b Compare February 20, 2026 14:11

lintingbin mentioned this pull request Feb 27, 2026

[AMORO-4044][amoro-format-iceberg] Fix SLF4J log format placeholder in TableEntriesScan #4102

Merged

2 tasks

Merge branch 'master' into fix/AMORO-4044-partition-mapping-bug

7f30d38

xxubai reviewed Mar 5, 2026

View reviewed changes

xxubai approved these changes Mar 6, 2026

View reviewed changes

xxubai merged commit 02f7c4c into apache:master Mar 6, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMORO-4044] Fix partition mapping bug in TableEntriesScan for tables with evolved PartitionSpec#4085

[AMORO-4044] Fix partition mapping bug in TableEntriesScan for tables with evolved PartitionSpec#4085
xxubai merged 2 commits intoapache:masterfrom
j1wonpark:fix/AMORO-4044-partition-mapping-bug

j1wonpark commented Feb 13, 2026 •

edited

Loading

Uh oh!

j1wonpark commented Feb 15, 2026

Uh oh!

lintingbin commented Feb 27, 2026

Uh oh!

j1wonpark commented Mar 5, 2026

Uh oh!

lintingbin commented Mar 5, 2026

Uh oh!

xxubai Mar 5, 2026

Uh oh!

j1wonpark Mar 5, 2026

Uh oh!

xxubai Mar 6, 2026

Uh oh!

xxubai Mar 5, 2026

Uh oh!

xxubai Mar 5, 2026

Uh oh!

xxubai Mar 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	builder.withPartition(projectPartition(spec, partition));
	builder.withPartition(PartitionUtil.coercePartition(unifiedPartitionType(), spec, partition));

Conversation

j1wonpark commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

Uh oh!

j1wonpark commented Feb 15, 2026

Uh oh!

lintingbin commented Feb 27, 2026

Uh oh!

j1wonpark commented Mar 5, 2026

Uh oh!

lintingbin commented Mar 5, 2026

Uh oh!

xxubai Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

j1wonpark Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

xxubai Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

xxubai Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

xxubai Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

xxubai Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

j1wonpark commented Feb 13, 2026 •

edited

Loading