[core] Introduce blob v2 by leaves12138 · Pull Request #7216 · apache/paimon

leaves12138 · 2026-02-05T08:11:56Z

Purpose

Blob v2 contains the following new features:

Write adaptive. Paimon could find out the descriptor or the blob itself you wrote, user does not need to set 'blob-as-descriptor' to true when write blob descriptor to paimon
Descriptor storage. User could set blob.stored-descriptor-fields = 'xxx' to store blob descriptor in normal (parquet/orc/avro rather than blob)file. It just store the descriptor as bytes, does not copy anything from descriptor to paimon table.

Tests

API and Format

Documentation

JingsongLi · 2026-02-10T15:21:42Z

paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java

        ByteBuffer buffer = ByteBuffer.allocate(totalSize);
        buffer.order(ByteOrder.LITTLE_ENDIAN);

+        buffer.putLong(MAGIC);


I think in order to be compatible with older versions, we should design Magic Numbers after the version number, and Linux kernel images are designed in this way.

JingsongLi · 2026-02-12T03:29:18Z

Due to compatibility issues with this PR, I suggest that we modify all relevant content in this PR, including:

Blob-descriptor V2 and write blob with adaptive match blob-descriptor.
Support blob-store-descriptor.
Modify PyPaimon to support *1 and *2

leaves12138 · 2026-02-14T07:54:45Z

Due to compatibility issues with this PR, I suggest that we modify all relevant content in this PR, including:

Blob-descriptor V2 and write blob with adaptive match blob-descriptor.

Support blob-store-descriptor.

Modify PyPaimon to support *1 and *2

OK

JingsongLi · 2026-02-26T03:50:35Z

paimon-api/src/main/java/org/apache/paimon/CoreOptions.java


+    @Immutable
+    public static final ConfigOption<String> BLOB_STORED_DESCRIPTOR_FIELDS =
+            key("blob.stored-descriptor-fields")


blob-descriptor-field

JingsongLi · 2026-02-26T03:58:18Z

paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRow.java

+        this(vectorizedColumnBatch, null, rowId);
+    }
+
+    public ColumnarRow(VectorizedColumnBatch vectorizedColumnBatch, UriReader uriReader) {


Do not modify constructors. Just use setFileIO.

JingsongLi · 2026-02-26T03:58:26Z

paimon-common/src/main/java/org/apache/paimon/data/columnar/ColumnarRow.java

        this.rowId = 0;
    }

+    public void setUriReader(UriReader uriReader) {


JingsongLi · 2026-02-26T04:03:11Z

paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java

+        byte version = buffer.get();
+        if (version == 1) {
+            try {
+                deserialize(bytes);


Remove codes for v1, only works for v2.

JingsongLi · 2026-02-26T04:03:19Z

paimon-common/src/main/java/org/apache/paimon/utils/UriReaderFactory.java


 /** A factory to create and cache {@link UriReader}. */
-public class UriReaderFactory {
+public class UriReaderFactory implements Serializable {


Remove this.

SparkRow may use this

discivigour · 2026-02-26T06:19:15Z

paimon-python/pypaimon/common/options/core_options.py

+        .with_description("Whether to return blob values as serialized BlobDescriptor bytes when reading.")
+    )
+
+    BLOB_STORED_DESCRIPTOR_FIELDS: ConfigOption[str] = (


Maybe It's better for BLOB_STORED_DESCRIPTOR_FIELDS and blob-descriptor-field to be the same.

discivigour · 2026-02-26T06:24:28Z

paimon-python/pypaimon/filesystem/local_file_io.py

    def write_blob(self, path: str, data: pyarrow.Table, blob_as_descriptor: bool, **kwargs):
        try:
+            # Kept for API compatibility. Write behavior is adaptive and does not depend on this flag.
+            _ = blob_as_descriptor


If the param blob_as_descriptor is not needed in this method, removing the param is better?

JingsongLi · 2026-02-26T08:22:08Z

paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java

    public static BlobDescriptor deserialize(byte[] bytes) {
        ByteBuffer buffer = ByteBuffer.wrap(bytes);
        buffer.order(ByteOrder.LITTLE_ENDIAN);
-


Remove this.

JingsongLi · 2026-02-26T08:22:24Z

paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java

        buffer.order(ByteOrder.LITTLE_ENDIAN);
-
        byte version = buffer.get();
        if (version != CURRENT_VERSION) {


V1 compatiblity and add tests.

JingsongLi · 2026-02-26T08:22:48Z

paimon-api/src/main/java/org/apache/paimon/types/BlobType.java

+     * fields.
+     */
+    public static Pair<RowType, RowType> splitBlob(
+            RowType rowType, Set<String> blobStoredDescriptorFields) {


blobDescriptorFields

JingsongLi · 2026-02-26T08:23:16Z

paimon-core/src/main/java/org/apache/paimon/append/AppendOnlyWriter.java

            boolean asyncFileWrite,
            boolean statsDenseStore,
            @Nullable BlobConsumer blobConsumer,
+            Set<String> blobStoredDescriptorFields,


blobDescriptorFields

JingsongLi · 2026-02-26T08:23:30Z

paimon-core/src/main/java/org/apache/paimon/append/MultipleBlobFileWriter.java

-            @Nullable BlobConsumer blobConsumer) {
-        RowType blobRowType = BlobType.splitBlob(writeSchema).getRight();
+            @Nullable BlobConsumer blobConsumer,
+            Set<String> blobStoredDescriptorFields) {


blobDescriptorFields

JingsongLi · 2026-02-26T08:26:00Z

paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java

+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;

 /** Tests for table with blob. */
 public class BlobTableTest extends TableTestBase {


Add a test to validate orc and avro.

JingsongLi · 2026-02-26T08:26:02Z

paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java

+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;

 /** Tests for table with blob. */
 public class BlobTableTest extends TableTestBase {


Add a test to validate orc and avro.

JingsongLi · 2026-02-26T08:26:42Z

paimon-python/pypaimon/table/row/blob.py

+    MAGIC = 0x424C4F4244455343  # "BLOBDESC"

    def __init__(self, uri: str, offset: int, length: int, version: int = CURRENT_VERSION):
+        if version != self.CURRENT_VERSION:


Remember to sync to python.

leaves12138 · 2026-02-26T10:04:16Z

Hi @JingsongLi, thanks for the review comments. I have addressed all of them in this PR and synchronized the related Java/Python changes accordingly. Could you please take another look when you have time?

This reverts commit 74ff427.

…b write test

JingsongLi · 2026-02-26T14:14:01Z

paimon-python/pypaimon/tests/blob_table_test.py

+        self.assertEqual(result.column('pic1').to_pylist()[0], pic1_data)
+        self.assertEqual(result.column('pic2').to_pylist()[0], pic2_data)
+
+    def test_blob_stored_descriptor_fields_rejects_non_descriptor_input(self):


all blob_stored_descriptor_fields to blob_descriptor_field

JingsongLi

+1

leaves12138 changed the title ~~[flink] [spark] Write blob with adaptive match blob-descriptor~~ [WIP] [flink] [spark] Write blob with adaptive match blob-descriptor Feb 5, 2026

leaves12138 changed the title ~~[WIP] [flink] [spark] Write blob with adaptive match blob-descriptor~~ [flink] [spark] Write blob with adaptive match blob-descriptor Feb 5, 2026

JingsongLi reviewed Feb 10, 2026

View reviewed changes

leaves12138 force-pushed the blob_descriptor_adaptive_deserialization branch from 0084314 to 58ab357 Compare February 11, 2026 05:25

leaves12138 requested a review from JingsongLi February 12, 2026 02:09

leaves12138 marked this pull request as draft February 14, 2026 01:59

leaves12138 force-pushed the blob_descriptor_adaptive_deserialization branch from c28fc68 to 226f6e8 Compare February 14, 2026 03:01

leaves12138 changed the title ~~[flink] [spark] Write blob with adaptive match blob-descriptor~~ [core] Introduce blob v2 Feb 14, 2026

[core] Introduce blob v2

428e913

leaves12138 force-pushed the blob_descriptor_adaptive_deserialization branch from a26004b to 428e913 Compare February 24, 2026 06:51

leaves12138 marked this pull request as ready for review February 24, 2026 07:03

Fix minus

a1f805b

JingsongLi reviewed Feb 26, 2026

View reviewed changes

leaves12138 added 5 commits February 26, 2026 13:53

Fix comment

458f514

Fix minus

8e604ed

Fix comment

6a34c85

Fix minus

32b7b8a

Fix minus

98362b7

discivigour reviewed Feb 26, 2026

View reviewed changes

Fix minus

73ba62c

JingsongLi reviewed Feb 26, 2026

View reviewed changes

leaves12138 added 5 commits February 26, 2026 17:17

Fix comment

69bb166

Fix comment

6769d2a

Fix comment

360ea64

Fix comment

4eabeef

[test] Adjust mixed blob storage mode assertion

43c8e17

leaves12138 added 3 commits February 26, 2026 18:05

[python] Fix blob descriptor v1 detection compatibility

74ff427

Revert "[python] Fix blob descriptor v1 detection compatibility"

8b3486b

This reverts commit 74ff427.

[python][test] Align blob descriptor API and stabilize concurrent blo…

9d76aa5

…b write test

JingsongLi reviewed Feb 26, 2026

View reviewed changes

JingsongLi approved these changes Feb 26, 2026

View reviewed changes

JingsongLi merged commit 4ca4544 into apache:master Feb 26, 2026
17 of 19 checks passed

Conversation

leaves12138 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Feb 12, 2026

Uh oh!

leaves12138 commented Feb 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leaves12138 commented Feb 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leaves12138 commented Feb 5, 2026 •

edited

Loading