Skip to content

fix: preserve non-empty theta v2 exact sketches - #189

Merged
tisonkun merged 1 commit into
mainfrom
codex/fix-theta-v2-empty-state
Aug 9, 2026
Merged

fix: preserve non-empty theta v2 exact sketches#189
tisonkun merged 1 commit into
mainfrom
codex/fix-theta-v2-empty-state

Conversation

@tisonkun

@tisonkun tisonkun commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

Fix legacy Theta serialization version 2 exact images being deserialized as logically empty when they contain retained entries.

The v2 exact branch now derives logical emptiness from num_entries == 0 instead of setting it unconditionally.

Closes #186.

Regression coverage

The new serialization compatibility tests construct valid v2 exact images using the current Theta family ID and default seed hash, then verify:

  • a non-empty image retains all entries,
  • is_empty() is false,
  • exact estimate and bounds equal the retained count,
  • entry iteration is unchanged,
  • a current-format round trip preserves the state, and
  • a zero-entry v2 exact image remains empty.

Relationship to other implementations

This restores agreement with the current C++ and Go legacy readers. Both treat a v2 exact image (preamble_longs == 2) as empty only when num_entries == 0:

Go retains compatibility fixtures identified as Java-generated v2 images. Current Java reads compact versions 3 and 4 rather than v2 directly, so the active legacy-reader behavior is represented by C++ and Go:

The fix changes no serialization bytes emitted by Rust; it only corrects the state reconstructed from legacy input.

Validation

  • cargo x prepare-testdata
  • cargo x check
  • cargo x test
  • cargo x lint

@tisonkun
tisonkun requested review from ZENOTME and a lite review from Copilot August 9, 2026 16:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes legacy Theta compact serialization v2 (“exact”, preamble_longs == 2) being deserialized as logically empty even when it contains retained entries, aligning Rust behavior with the C++ and Go legacy readers and preventing downstream set-operation short-circuiting.

Changes:

  • Fix v2 exact deserialization to set empty based on num_entries == 0 instead of unconditionally true.
  • Add regression tests that construct valid v2 exact images and assert non-empty behavior, estimate/bounds correctness, iteration stability, and current-format round-trip preservation.
  • Add a regression test ensuring zero-entry v2 exact images remain logically empty.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
datasketches/src/thetafamily/theta/sketch.rs Corrects v2 exact logical-emptiness reconstruction by tying empty to num_entries.
datasketches/tests/serde_tests/theta.rs Adds compatibility tests and a helper to synthesize v2 exact images for regression coverage.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread datasketches/tests/serde_tests/theta.rs

@ZENOTME ZENOTME left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Maybe we refine test later.


use crate::serialization_test_data;

fn serialize_v2_exact(entries: &[u64]) -> Vec<u8> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure there are better way to provide a v2 serialize snapshot instead of construct by hand here. 🤔

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

v2 serialize snapshot

Maybe. But existing C++/Java/Go impls do not provide one. We may provide one when implementing #176 , and if you'd like to, you can go to other impls to ask them to provide one and update https://github.com/apache/datasketches-tck.

@tisonkun
tisonkun marked this pull request as draft August 9, 2026 16:44
@tisonkun
tisonkun marked this pull request as ready for review August 9, 2026 18:38
@tisonkun
tisonkun merged commit 5a1ab56 into main Aug 9, 2026
11 checks passed
@tisonkun
tisonkun deleted the codex/fix-theta-v2-empty-state branch August 9, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Theta v2 exact sketches deserialize as logically empty

3 participants