HDDS-13217. Test to compare OM snapshot checkpoint metadata to live OM for a bucket#10235
Open
arunsarin85 wants to merge 2 commits into
Open
HDDS-13217. Test to compare OM snapshot checkpoint metadata to live OM for a bucket#10235arunsarin85 wants to merge 2 commits into
arunsarin85 wants to merge 2 commits into
Conversation
…adata to live OM for a bucket
adoroszlai
reviewed
May 11, 2026
Comment on lines
+104
to
+105
| cluster = MiniOzoneCluster.newBuilder(conf).setNumDatanodes(3).build(); | ||
| cluster.waitForClusterToBeReady(); |
Contributor
There was a problem hiding this comment.
New cluster creation takes ~30 seconds. If the test is not destructive, can it be added to an existing test class? Or can it use shared cluster? (see HDDS-12183 for example)
Contributor
Author
There was a problem hiding this comment.
Added a patch .
adoroszlai
changed the title
jojochuang
requested review from
SaketaChalamchala,
Copilot,
jojochuang,
sadanand48 and
smengcl
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an integration test to validate that OM snapshot on-disk RocksDB checkpoints contain the same bucket-scoped metadata as the live OM DB across bucket layouts.
Changes:
- Introduces
TestOmSnapshotCheckpointDbContentwhich stands up a MiniOzoneCluster and exercises OBJECT_STORE, FILE_SYSTEM_OPTIMIZED, and LEGACY buckets. - Creates snapshots, waits for checkpoint materialization, opens the checkpoint DB read-only, and compares bucket-prefixed rows across selected OM tables.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } | ||
| try (KeyValueIterator<String, V> it = table.iterator(prefix)) { | ||
| while (it.hasNext()) { | ||
| KeyValue<String, V> kv = it.next(); |
Comment on lines
+173
to
+179
| String currentPath = | ||
| OmSnapshotManager.getSnapshotPath(conf, snapshotInfo, 0) | ||
| + OM_KEY_PREFIX + "CURRENT"; | ||
| GenericTestUtils.waitFor(() -> new File(currentPath).exists(), 1000, 120_000); | ||
|
|
||
| RocksDBCheckpoint checkpoint = new RocksDBCheckpoint( | ||
| Paths.get(OmSnapshotManager.getSnapshotPath(conf, snapshotInfo, 0))); |
Comment on lines
+222
to
+224
| String prefix = prefixes.getTablePrefix(tableName); | ||
| assertEquals(readPrefix(live, prefix), readPrefix(checkpoint, prefix), | ||
| tableName); |
Contributor
SaketaChalamchala
left a comment
SaketaChalamchala
left a comment
There was a problem hiding this comment.
The current patch seems to compare the contents of a snapshot and live OM.
The intent of the JIRA seems to be to verify the integrity of snapshots after defrag iterations.
Ex.,
Setup - Take snapshots : S1(at T1) S2(at T2) S3(at T3)
Test#1:
Run Defrag Iteration#1: S1' = compact(S1) S2' = S1' + diff(S1, S2) S3' = S1' + diff(S1, S2) + diff(S2, S3)
Verify contents: S1' == S1 S2' == S2 S3' == S3
Test#2:
Delete S2
Run Defrag Iteration#2: S2'' = compact(S2') S3'' = S2'' + diff(S2, S3)
Verify contents: S2'' == S2 S3'' == S3
cc @smengcl could you confirm?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds a new integration test class,TestOmSnapshotCheckpointDbContent. It builds a small MiniOzone cluster, writes keys for OBJECT_STORE, FILE_SYSTEM_OPTIMIZED, and LEGACY buckets, creates a snapshot, waits for the on-disk checkpoint to appear, opens that checkpoint read-only, and compares bucket-scoped rows between the live OM metadata manager and the checkpoint for the tables the test covers.
Please describe your PR in detail:
Snapshot correctness depends on the OM checkpoint on disk reflecting the same bucket metadata as the active OM for the data paths clients care about.
For each bucket layout, the flow is: create a volume and bucket, write a few keys, call createSnapshot, load SnapshotInfo from the live OM, wait until the checkpoint CURRENT file exists, open the checkpoint with RocksDBCheckpoint and OmMetadataManagerImpl.createCheckpointMetadataManager, then scan matching key prefixes on both sides.
Compared tables include volume and bucket rows, the key table for object store and legacy layouts, file and directory tables for FSO.
The new snapshot row in snapshotInfoTable commits in the same batch as the checkpoint step, so it appears on the live OM after commit but not inside the checkpoint taken mid-batch. After the checkpoint, OM also scrubs deleted and snapshot-renamed prefix ranges on the active DB, so live and checkpoint diverge there on purpose. Those tables are not part of the equality check.
Three datanodes for RATIS/THREE, snapshot defrag disabled for stability, and a null check on SnapshotInfo for clearer failures.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13217
How was this patch tested?
Locally with Maven, scoped to the new class:
mvn -pl hadoop-ozone/integration-test -Dtest=TestOmSnapshotCheckpointDbContent -DfailIfNoTests=false test
