EXPORT PARTITION writes unreadable Iceberg manifest list file if existing file has key_metadata field

[xieandrew]

Describe the bug
An EXPORT PARTITION to an Iceberg table with existing snapshots written by Ice generates a new snapshot with an invalid manifest list avro file. This file cannot be read by ClickHouse or avro-tools. The cause seems to be a missing key_metadata section in the Avro schema declaration when the manifest file records include a key_metadata value.

key_metadata is an optional field in the Iceberg spec.

The existing manifest list file (generated by Ice) contains key_metadata in the schema. It seems that EXPORT PARTITION writes a new manifest list by copying the existing manifest records (which include "key_metadata":null) but doesn't write an object for this field in the Avro schema declaration.

To Reproduce
Steps to reproduce the behavior:

1. Ensure an instance of ice-rest-catalog is running.

2. Create an Iceberg table and insert data with Ice (pointing to the running ice-rest-catalog).

ice create-table --partition='[{"column":"variety","transform":"identity"}]' --schema-from-parquet=iris.parquet test.iris_export

ice insert -p test.iris_export iris.parquet

Inspecting the manifest list file with avro-tools tojson snap-1698375957403334217-1-a51acdcb-abcf-4edb-a52e-85ba2f847b8c.avro shows this as the content:

{
  "manifest_path" : "s3://bucket-name/test/iris_export/metadata/a51acdcb-abcf-4edb-a52e-85ba2f847b8c-m0.avro",
  "manifest_length" : 7727,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1,
  "min_sequence_number" : 1,
  "added_snapshot_id" : 1698375957403334217,
  "added_files_count" : 3,
  "existing_files_count" : 0,
  "deleted_files_count" : 0,
  "added_rows_count" : 150,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "Setosa"
      },
      "upper_bound" : {
        "bytes" : "Virginica"
      }
    } ]
  },
  "key_metadata" : null
}

This includes a "key_metadata": null value at the bottom. Inspecting the Avro schema with avro-tools getschema shows the correct schema object at the end of the array:

{
    "name" : "key_metadata",
    "type" : [ "null", "bytes" ],
    "doc" : "Encryption key metadata blob",
    "default" : null,
    "field-id" : 519
}]

3. In ClickHouse, create a database ice connected to the running ice-rest-catalog:

CREATE DATABASE ice
  ENGINE = DataLakeCatalog('http://ice-rest-catalog:5000')
  SETTINGS catalog_type = 'rest',
    auth_header = 'Authorization: Bearer your-auth-token', 
    warehouse = 's3://your-warehouse';

You should see the table created by Ice in step 2 using SHOW TABLES FROM ice.

4. Create a MergeTree table with the same partitioning as the Iceberg table to export from and insert test data:

CREATE TABLE default.iris_export
(
    `sepal.length` Float64,
    `sepal.width` Float64,
    `petal.length` Float64,
    `petal.width` Float64,
    `variety` String
)
ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/iris_export/{shard}', '{replica}')
PARTITION BY variety

INSERT INTO default.iris_export VALUES (1.1, 1.2, 1.3, 1.4, 'Exported'), (2.1, 2.2, 2.3, 2.4, 'Exported')

5. Find the partition id and run EXPORT PARTITION from the MergeTree table to the Iceberg table:

Find partition id:

SELECT
    partition_id,
    sum(rows)
FROM system.parts
WHERE (database = 'default') AND (`table` = 'iris_export') AND active
GROUP BY partition_id
ORDER BY partition_id ASC

Use the found partition_id value in the export query:

ALTER TABLE default.iris_export EXPORT PARTITION ID 'bb506a317a71d534b654e6030ef81336' TO TABLE ice.`test.iris_export`

6. Wait for the export to complete, then find the generated manifest list file name for the new snapshot and inspect it.

Using avro-tools:

avro-tools tojson snap-349868014-2-8a50d555-866d-4409-a569-385ad891c48a.avro

Exception in thread "main" org.apache.avro.AvroRuntimeException: java.io.IOException: Block read partially, the data may be corrupt
        at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:237)
        at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:96)
        at org.apache.avro.tool.Main.run(Main.java:67)
        at org.apache.avro.tool.Main.main(Main.java:56)
Caused by: java.io.IOException: Block read partially, the data may be corrupt
        at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:222)
        ... 3 more

Using ice CLI:

ice describe-metadata s3://bucket-name/test/iris_export/metadata/v2-27d9420f-3662-441b-ac1e-36d3a4da113c.metadata.json --manifests

2026-08-06 16:02:05 [main/384181] WARN c.a.i.c.i.c.DescribeMetadata > Failed to read manifest list: java.io.IOException: Block read partially, the data may be corrupt
---
manifests: []

Some exports allow ClickHouse to read the table still, but I encountered this error with a different export:

SELECT count()
FROM ice.`export.table_name`

Received exception from server (version 26.3.13):
Code: 117. DB::Exception: Received from clustername.envname.altinity.cloud:9440. DB::Exception: Cannot read Iceberg avro manifest file 's3://bucket-name/export/table_name/metadata/snap-1631653436-2-8969816f-502e-4aec-89aa-10d0d60d768c.avro': Invalid value for bool. (INCORRECT_DATA)

Expected behavior
ClickHouse should write a valid manifest list Avro file readable by ClickHouse and external tools.

The manifest list file (snap-*-.avro) should include key_metadata in the Avro schema declaration if the existing manifest list file contains it:

{
    "name" : "key_metadata",
    "type" : [ "null", "bytes" ],
    "doc" : "Encryption key metadata blob",
    "default" : null,
    "field-id" : 519
}

This issue may extend to other optional Iceberg manifest list fields that ClickHouse doesn't write in the Avro schema declaration, but other fields have not been tested.

Key information
Provide relevant runtime details.

• Project Antalya 26.3.13.20001
• Altinity Cloud
• AWS S3
• ice-rest-catalog