Skip to content

Cherry-pick cloud snapshot related PRs to branch-4.0 #59470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
w41ter wants to merge 78 commits into
base: branch-4.0
Choose a base branch
from
Open

Cherry-pick cloud snapshot related PRs to branch-4.0 #59470

w41ter wants to merge 78 commits into from

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Dec 30, 2025

No description provided.

Yukang-Lian and others added 30 commits December 30, 2025 11:10
@Yukang-Lian @w41ter
[improve](snapshot) Modify some snapshot interface (apache#56393)
1f90646
@w41ter
[feat](cloud) Add clone chain reader (apache#56401)
e930fe8
@w41ter
[fix](cloud) Read delete bitmap v2 via clone chain reader (apache#56461)
0896951
@w41ter
[improve](cloud) unify snapshot properties to enum name (apache#56337)
c59a057
@w41ter
[fix](cloud) Fill schema table cluster_snapshots (apache#56575)
3e11767
@mymeiyi @w41ter
[feat](snapshot) recycle support abort multipart upload (apache#56628)
19805e1
@w41ter
[chore](cloud) Add enable_multi_version_status & enable_cluster_snaps…
3115dfa 
…hot config (apache#56614)
@wyxxxcat @w41ter
[feat](cloud) Add check mvcc meta rowset key for checker (apache#56463)
56c0792
@deardeng @w41ter
[Fix](cloud) Fix cloud compile err (apache#56686)
924cd15 
Fix cloud compile err, due to apache#56628
```
/mnt/disk2/doris/apache_doris/cloud/src/recycler/azure_obj_client.h:29:7: error: abstract class is marked 'final' [-Werror,-Wabstract-final-class]
   29 | class AzureObjClient final : public ObjStorageClient {
      |       ^
/mnt/disk2/doris/apache_doris/cloud/src/recycler/obj_storage_client.h:106:35: note: unimplemented pure virtual method 'abort_multipart_upload' in 'AzureObjClient'
  106 |     virtual ObjectStorageResponse abort_multipart_upload(ObjectStoragePathRef path,
      |                                   ^
```
@wyxxxcat @w41ter
[chore](checker) Add get_all_tablet_ids for MetaReader skeleton (ap…
3fe92b7 
…ache#56888)
@Yukang-Lian @w41ter
[Enhancement](Snapshot) Add some snapshot interface (apache#56953)
80e2d73
@mymeiyi @w41ter
[feat](snapshot) fix Unknown snapshot status when select snapshots (a…
25b650c 
…pache#56726)
@w41ter
[chore](cloud) Add decode_*_key (apache#57008)
eb907df
@w41ter
[chore](cloud) expose split_tablet_stats_into_load_and_compact_parts (a…
a2bab83 
…pache#57009)
@w41ter
[feat](cloud) Add snapshot data migrator & chain compactor skeleton (a…
879f0b0 
…pache#56939)
@w41ter
[fix](cloud) Fix /get_snapshot_property result (apache#57067)
c2df71d
@w41ter
[fix](cloud) Return error if migrate to versioned keys/compact snapsh… (
5923373 
apache#57066)

…ot chains is not implemented
@w41ter
[fix](cloud) Fix /set_snapshot_property property names (apache#57099)
4a2a30c
@w41ter
[chore](cloud) support notifing itself to refresh instance (apache#57107
d229066 
)
@gavinchou @w41ter
[chore](cloud) Make key info declaration more clear (apache#57034)
d0bf7d0
@wyxxxcat @w41ter
[chore](checker) Add skeleton for mvcc meta checker (apache#57174)
ac850ec
@Yukang-Lian @w41ter
[chore](cloud) Add find_derived_instance_ids function (apache#57274)
2898755
@w41ter
[chore](cloud) Disable auto snapshot by default (apache#57285)
e4e1b7a
@w41ter
[feat](cloud) Support Transaction::watch_key (apache#57408)
5b85794
@Yukang-Lian @w41ter
[Feat](Instance) Cascade AK/SK to derived instances (apache#57379)
17b48a6
@w41ter
[feat](cloud) Add instance updates watcher (apache#57435)
c605ada 
to refresh instance info more accurately
@w41ter
[chore](cloud) Add some utility functions (apache#57467)
48e60b5
@w41ter
[fix](cloud) Avoid recycle orphan partitions if data migrating is not…
0ed73d3 
… finished (apache#57550)
@w41ter
[feat](cloud) Ignore the staled clusters of the rollbacked instance (a…
8ceba18 
…pache#57478)

during the instance refreshing
@w41ter
[fix](cloud) Remove global snapshot data migrating context (apache#57553
454b438 
)
w41ter and others added 15 commits December 30, 2025 11:22
@w41ter
[fix](cloud) correct the min_read_version of the CloneChainReader (ap…
1fdea8d 
…ache#59086)
@wyxxxcat @w41ter
[chore](checker) Add resource_mgr for checker (apache#58954)
c56d94a
@mymeiyi @w41ter
[feat](snapshot) fix recycle deleted instance (apache#59134)
890dc2c
@w41ter
[improve](cloud) Allow to control txn lazy commit advancing during re…
275af26 
…ads (apache#59196)
@w41ter
[improve](cloud) TxnLazyCommitter do not block bthreads (apache#59216)
ff53c67
@w41ter
[improve](cloud) Get visible version explicitly wait for pending txns (…
2d27701 
…apache#59213)
@w41ter
[improve](cloud) Shuffle the partitions to reduce committing conflicts (
d01316c 
apache#59212)
@mymeiyi @w41ter
[feat](snapshot) calculate snapshot data size (apache#59168)
0f99074
@wyxxxcat @w41ter
[chore](partition) Write versioned partition key when commit index (a…
93d0460 
…pache#59078)
@Yukang-Lian @w41ter
[Feature](Recycler) Batch delete optimization for versioned tablet re…
bec4c39 
…cycling (apache#59247)
@w41ter
[fix](cloud) fix tablet stats for versioned keys (apache#59193)
c275a73
@w41ter
[improve](cloud) Txn lazy committer commit partitions in parallel (ap…
13d6757 
…ache#59291)
@w41ter
[feat](cloud) Support fdb multi version clients (apache#59360)
051369e
@Yukang-Lian @w41ter
[fix](txn) fix clean_txn_label racing with recycler in versioned writ…
42e7d50 
…e mode (apache#59443)
@mymeiyi @w41ter
[feat](snapshot) record upload id in snapshot (apache#56626)
f26307c
@w41ter w41ter requested a review from yiguolei as a code owner December 30, 2025 03:32
@Thearas
Copy link
Contributor

Thearas commented Dec 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

BE UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.36% (18670/34989)
Line Coverage 39.12% (172934/442025)
Region Coverage 33.79% (133622/395466)
Branch Coverage 34.77% (57782/166165)

@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

1 similar comment
@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

@w41ter
Copy link
Contributor Author

w41ter commented Dec 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

BE UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.35% (18667/34989)
Line Coverage 39.13% (172958/441997)
Region Coverage 33.78% (133574/395426)
Branch Coverage 34.78% (57785/166153)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@w41ter @Thearas @doris-robot @Yukang-Lian @mymeiyi @wyxxxcat @deardeng @gavinchou