fix(core): stop serving stale row id sequences after overwrite

[wjones127]

Fragment ids were reused across generations of a dataset: WriteMode::Overwrite restarted them at 0. The cached row id sequence for a fragment was looked up by fragment id alone, so with a shared Session every reader after an overwrite was handed the previous generation's sequence for the reused id.

That silently corrupts stable row ids: row ids are reported for rows that no longer exist, row counts disagree with the manifest, compaction rechunks more ids than the fragments physically hold, and the same id can look live in two fragments at once.

This PR fixes both halves:

• Fragment ids are now a high water mark that never rewinds, including on overwrite. This is the invariant from discussion Non-linear history and `max_fragment_id` and `next_row_id` #5540 — "we need fragment_id and row_id to be globally unique across the table history" — which fix: restore decrease max_fragment_id in manifest #5554 already established for restore. Overwrite was the remaining hole.
• The row id sequence cache key now carries the fragment's row id metadata. Ids can still be reused by paths that don't consult a previous manifest at all: drop a dataset and create a new one at the same URI (RowIdSequenceKey cache key lacks a dataset version/generation — stale hit across drop + recreate at the same URI #7645) and ids restart at 0. Datasets written before this change also keep duplicate ids in their history. Content-keying is correct regardless of id allocation: identical metadata implies identical sequence bytes, so a hit is always a valid hit, while a rewritten fragment misses. Operations that leave row ids alone (deletes, added columns) keep hitting the cache.

A version-scoped key was proposed for this same spot in #5508 and closed in favour of making fragment ids unique; keying on the manifest version would also fix the overwrite case but would discard the cache on every commit.

This does not fully close #7645: RowIdIndexKey and ManifestKey are still keyed on (uri, version), and version restarts at 1 for a dataset recreated at the same URI.

Breaking changes

Fragment ids no longer restart at 0 after an overwrite — the first fragment written by an overwrite now takes the next id after the dataset's high water mark. Code that assumes a specific fragment id after an overwrite (e.g. dataset.get_fragment(0)) needs to read the ids from the manifest instead.

Fixes #8075

[codecov]

Codecov Report

❌ Patch coverage is 88.88889% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/session/caches.rs	53.33%	7 Missing ⚠️
rust/lance/src/dataset/rowids.rs	96.96%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

I independently wrote this same fix in #8080 (now closed as a duplicate — this PR was first and its tests are better). Two things from that work worth folding in here.

Prior history the reviewer will want. This key has been reported three times from three different directions, and a previous fix in this exact spot was closed in favour of a different design:

• RowIdSequenceKey cache key lacks a dataset version/generation — stale hit across drop + recreate at the same URI #7645 (open) — same root cause, reached via drop + recreate at the same URI rather than overwrite. This PR fixes the row-id-sequence half of it. Worth noting that content-keying is safe for that trigger rather than merely lucky: an identical key implies identical sequence bytes, so a recreated dataset either misses or hits an entry that is correct anyway. The rest of RowIdSequenceKey cache key lacks a dataset version/generation — stale hit across drop + recreate at the same URI #7645 stays open, since RowIdIndexKey and ManifestKey are still (uri, version)-keyed and still alias across drop + recreate.
• fix(cache): scope RowIdSequence cache by version #5508 (closed) — scoped this key by version, triggered by restore. Closed after @wjones127 redirected it toward making fragment_id globally unique instead: "This seems easy to accomplish with Overwrite (just don't reset the max_fragment_id)." That led to discussion Non-linear history and `max_fragment_id` and `next_row_id` #5540, and fix: restore decrease max_fragment_id in manifest #5554 fixed the restore half. The Overwrite half was never done — transaction.rs:1854 still hard-resets fragment_id to 0 for Operation::Overwrite.

So this PR is the lightweight route that #5508 was steered away from. That seems right to me — content-keying is correct independently of whether fragment ids are ever made unique, and it also covers drop + recreate, which the max_fragment_id change would not. But it's worth being explicit that it doesn't close #5540, and that other fragment-id-keyed caches remain exposed to id reuse.

One suggestion on write_key. meta_discriminator() collapses the content to a 64-bit DefaultHasher digest before it reaches the KeyBuilder, which then hashes it into a blake3-128 key. That caps the physical key's effective collision resistance at the inner 64-bit hash (~32-bit birthday bound) and, per the KeyBuilder docs, gives up the encoding it asks for: "In-tree hot paths override this with typed, allocation-free field encoding." Writing the fields directly costs nothing extra:

fn write_key(&self, builder: &mut KeyBuilder) {
    builder.write_u64(self.fragment_id);
    match self.row_id_meta {
        RowIdMeta::Inline(data) => {
            builder.write_variant(0);
            builder.write_bytes(data);
        }
        RowIdMeta::External(file) => {
            builder.write_variant(1);
            builder.write_str(&file.path);
            builder.write_u64(file.offset);
            builder.write_u64(file.size);
        }
    }
}

meta_discriminator() is still needed for the string key(), so this is additive. Not a correctness issue at realistic scale — take it or leave it.

[wjones127]

Both suggestions folded in.

write_key now encodes the metadata fields directly into the KeyBuilder instead of collapsing them to a 64-bit digest first; meta_digest() remains only for the human-readable key() string. The prior history (#5508, #5540, #5554, #7645) is now in the description.

The overwrite half of #5540 is also in this PR: fragment ids no longer restart at 0 on overwrite, which is what #5508 was steered toward and what #5554 already did for restore. That is why the regression test for the key itself now lives in caches.rs — with ids unique across overwrite, the dataset-level tests no longer exercise a colliding id, but the key must still discriminate for the drop + recreate path in #7645.