Summary
EXPORT PARTITION into Iceberg writes Parquet data files without Iceberg field-ids. ClickHouse-created IcebergS3 destinations also do not set table property schema.name-mapping.default.
Strict Iceberg readers (e.g. PyIceberg) then fail when scanning exported data:
ValueError: Parquet file does not have field-ids and the Iceberg table does not have 'schema.name-mapping.default' defined
This blocks the intended Hybrid / tiered-storage workflow where cold data is exported to Iceberg and read by external engines.
Observed on Antalya 26.3.13.20001.
Expected
After EXPORT PARTITION to an Iceberg destination, an external Iceberg reader (PyIceberg, Spark, etc.) should be able to scan the committed snapshot and see the exported rows. Something like:
- exported Parquet embeds Iceberg field-ids matching the table schema, and/or
- ClickHouse sets
schema.name-mapping.default on Iceberg tables it creates / commits to
(Iceberg column projection: https://iceberg.apache.org/spec/#column-projection)
Actual
- EXPORT Parquet has no field-ids in the file schema.
IcebergS3 tables created by ClickHouse do not get schema.name-mapping.default.
- Even when the destination was created via PyIceberg (proper Iceberg schema in the catalog), EXPORT still writes Parquet without field-ids, so PyIceberg still fails unless the table property is set separately.
Reproduce
- Create MergeTree source with a partition; insert rows.
- Create Iceberg destination (
IcebergS3 or catalog-backed).
ALTER TABLE … EXPORT PARTITION … TO TABLE <iceberg> (with write_full_path_in_iceberg_metadata = 1 so StaticTable can open manifests).
- Load the table with PyIceberg and
table.scan().to_arrow().
PyIceberg raises the ValueError above during schema resolution.
Summary
EXPORT PARTITIONinto Iceberg writes Parquet data files without Iceberg field-ids. ClickHouse-createdIcebergS3destinations also do not set table propertyschema.name-mapping.default.Strict Iceberg readers (e.g. PyIceberg) then fail when scanning exported data:
This blocks the intended Hybrid / tiered-storage workflow where cold data is exported to Iceberg and read by external engines.
Observed on Antalya
26.3.13.20001.Expected
After
EXPORT PARTITIONto an Iceberg destination, an external Iceberg reader (PyIceberg, Spark, etc.) should be able to scan the committed snapshot and see the exported rows. Something like:schema.name-mapping.defaulton Iceberg tables it creates / commits to(Iceberg column projection: https://iceberg.apache.org/spec/#column-projection)
Actual
IcebergS3tables created by ClickHouse do not getschema.name-mapping.default.Reproduce
IcebergS3or catalog-backed).ALTER TABLE … EXPORT PARTITION … TO TABLE <iceberg>(withwrite_full_path_in_iceberg_metadata = 1so StaticTable can open manifests).table.scan().to_arrow().PyIceberg raises the
ValueErrorabove during schema resolution.