Skip to content

EXPORT PARTITION: Parquet lacks Iceberg field-ids (PyIceberg cannot scan without name-mapping) #2161

Description

@DimensionWieldr

Summary

EXPORT PARTITION into Iceberg writes Parquet data files without Iceberg field-ids. ClickHouse-created IcebergS3 destinations also do not set table property schema.name-mapping.default.

Strict Iceberg readers (e.g. PyIceberg) then fail when scanning exported data:

ValueError: Parquet file does not have field-ids and the Iceberg table does not have 'schema.name-mapping.default' defined

This blocks the intended Hybrid / tiered-storage workflow where cold data is exported to Iceberg and read by external engines.

Observed on Antalya 26.3.13.20001.

Expected

After EXPORT PARTITION to an Iceberg destination, an external Iceberg reader (PyIceberg, Spark, etc.) should be able to scan the committed snapshot and see the exported rows. Something like:

  1. exported Parquet embeds Iceberg field-ids matching the table schema, and/or
  2. ClickHouse sets schema.name-mapping.default on Iceberg tables it creates / commits to

(Iceberg column projection: https://iceberg.apache.org/spec/#column-projection)

Actual

  • EXPORT Parquet has no field-ids in the file schema.
  • IcebergS3 tables created by ClickHouse do not get schema.name-mapping.default.
  • Even when the destination was created via PyIceberg (proper Iceberg schema in the catalog), EXPORT still writes Parquet without field-ids, so PyIceberg still fails unless the table property is set separately.

Reproduce

  1. Create MergeTree source with a partition; insert rows.
  2. Create Iceberg destination (IcebergS3 or catalog-backed).
  3. ALTER TABLE … EXPORT PARTITION … TO TABLE <iceberg> (with write_full_path_in_iceberg_metadata = 1 so StaticTable can open manifests).
  4. Load the table with PyIceberg and table.scan().to_arrow().

PyIceberg raises the ValueError above during schema resolution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    antalyabugSomething isn't workingexport partitionIssues related to export partition functionalityiceberg writesAntalya Roadmap: Iceberg Writes

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions