[feat](fe) Support Cloud partition inverted format rollout - #66235
[feat](fe) Support Cloud partition inverted format rollout#66235hoshinojyunn wants to merge 1 commit into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
hoshinojyunn
changed the title
|
run buildall |
hoshinojyunn
changed the title
TPC-H: Total hot run time: 29746 ms |
TPC-DS: Total hot run time: 177881 ms |
ClickBench: Total hot run time: 25.08 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
airborne12
left a comment
airborne12
left a comment
There was a problem hiding this comment.
Two blocking lifecycle and rolling-upgrade issues are called out inline.
| TInvertedIndexFileStorageFormat invertedIndexFileStorageFormat = | ||
| origIdToInvertedIndexFileStorageFormat.get(origPartId); | ||
| if (invertedIndexFileStorageFormat != null) { | ||
| idToInvertedIndexFileStorageFormat.put(newPartId, invertedIndexFileStorageFormat); |
There was a problem hiding this comment.
Blocking: this remaps the partition-owned format in FE metadata, but CloudRestoreJob.createReplicas() still builds every restored Cloud tablet with localTbl.getInvertedIndexFileStorageFormat() and localTbl.getBaseSchemaVersion(). After restoring a mixed V2/V3 table, PartitionInfo can therefore say that a partition is V2 while its new TabletSchema is created from the current table-level V3 format/schema. Please pass the restored partition's resolved format and corresponding backup schema/version into createTabletMetaBuilder() so the physical tablets and partition metadata are restored consistently. This needs coverage for both full-table restore and partition-only restore with mixed V2/V3 partitions.
| TAddOrDropPartitionsResult result; | ||
| try { | ||
| result = masterCallWithRetry(client -> client.addOrDropPartitions(request), | ||
| result = masterCallWithRetry(client -> client.addPartitionsForInsertOverwrite(request), |
There was a problem hiding this comment.
Blocking: this introduces a new Thrift method without a rolling-upgrade compatibility path. During an FE rolling upgrade, a new client can call an old FE that does not implement addPartitionsForInsertOverwrite, resulting in TApplicationException.UNKNOWN_METHOD. Retrying the same method cannot recover, so INSERT OVERWRITE through a remote Doris catalog will fail until every target FE has been upgraded.
Please add explicit capability/version negotiation, or evolve the existing addOrDropPartitions request with an optional source-preserving flag and gate its use on server support. A blind fallback to the old method is not safe because it would create the temporary partition from the current table schema instead of preserving the source partition's physical schema. Please also add a new-client/old-server compatibility test.
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
| return partition; | ||
| } | ||
|
|
||
| public OlapFile.TabletMetaCloudPB getTabletMeta(long tabletId) throws DdlException { |
There was a problem hiding this comment.
[major/P1] Avoid issuing one synchronous Meta Service RPC per partition while holding the table write lock.
collectPartitionInvertedIndexFileStorageFormats() performs O(partition_count) sequential GetTablet RPCs under olapTable.writeLock. For tables with thousands of partitions, the ALTER can hold the table metadata lock for a long time. Any RPC failure aborts the operation before journaling, and retrying starts the full scan again. Meta Service rate limiting only turns this into latency or failure and does not make the operation scalable. Please batch the tablet-meta reads and perform network I/O outside the table write lock, then reacquire the lock and validate that the table/partition metadata has not changed before committing.
### What problem does this PR solve? Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. ### Release note Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. ### Check List (For Author) - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
3230373 to
d4eedd1
Compare
4 participants
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Cloud tables previously used one table-level
inverted_index_storage_format, so an online V2-to-V3 migration required all partitions to move together. This prevents a gradual rollout and makes it impossible to retain the correct tablet schema for old partitions while creating new partitions with a new format.This PR adds the Cloud-only table property
partition.inverted_index_storage_format. It is the format default for partitions created after the property is set. The property supports V2 and V3; V1 is rejected. Existing partitions and their tablet metas are not rewritten.Design:
PartitionInfofor every Cloud partition. The value is durable in journal/replay and is transferred through partition replacement, recycle/recover, add-partition-like, insert overwrite, and truncate flows.CloudSchemaChangeJobV2, and creates each partition's base shadow tablet with the matching format and schema version. Non-base indexes keep their normal schema-change behavior.SHOW PARTITIONS,SHOW PARTITION <id>,SHOW PARTITION ID, and the partitions TVF exposeInvertedIndexStorageFormatfromPartitionInfo; they no longer need to read tablet meta from MetaService for this value.Example SQL and expected effect:
Selected output:
Both formats remain queryable during the rollout:
The query returns both rows.
TRUNCATE TABLE ... PARTITION p_oldandINSERT OVERWRITE ... PARTITION(p_old)continue to create V2 tablet metas forp_old, even after the default changes to V3. A subsequentMODIFY COLUMNcreates distinct base shadow schema versions for active V2 and V3 partitions, then preserves the format/version pairing when committing the schema change.Release note
Cloud OLAP tables can roll out inverted-index file storage formats by partition. New partitions can use V3 while existing V2 partitions remain online and queryable.
Check List (For Author)
Feature and Test Coverage
MATCH_ANYqueries return rows from every partition.test_partition_cloud_inverted_index_storage_format_rolloutpartition.inverted_index_storage_format=V3creates its initial partition with V3.test_partition_cloud_inverted_index_storage_format_rollout; FE creation/property teststest_partition_cloud_inverted_index_storage_format_rolloutSHOW PARTITIONS,SHOW PARTITION <id>, and tablet meta report V2/V3 for range, list, auto, and dynamic partitions.test_cloud_show_inverted_index_storage_format; FEShowPartitionIdCommandTest, partitions TVF output testTRUNCATE TABLE/ partition replacementtest_partition_cloud_inverted_index_storage_format_recycle_truncateand rollout suite; FEOlapTableTest.testReplacePartitionPreservesInvertedIndexStorageFormatINSERT OVERWRITEtest_partition_cloud_inverted_index_storage_format_rollout; FE Cloud catalog and FrontendService teststest_partition_cloud_inverted_index_storage_format_recycle_truncate; FECatalogRecycleBinTest.testRecoverPartitionMODIFY COLUMN; shadow base and rollup indexes are validated while the job is RUNNING.test_partition_cloud_inverted_index_storage_format_rollout; FECloudSchemaChangeJobV2TestExecuted tests:
Changed FE test classes: 70 tests passed.
Targeted
OlapTableTestandCatalogRecycleBinTest: 41 tests passed.Cloud Docker regressions:
test_cloud_show_inverted_index_storage_formattest_partition_cloud_inverted_index_storage_format_recycle_truncatetest_partition_cloud_inverted_index_storage_format_rolloutBehavior changed:
partition.inverted_index_storage_format; it affects only subsequently created partitions and exposes the resolved format in partition metadata commands.Does this need documentation?
Check List (For Reviewer who merge this PR)