timely-util: UnloadChunk, bulk probe extraction for chunk batches - #37954
Merged
Conversation
UnloadChunk is the bulk-read capability for Chunk families: look up a sorted set of keys in a chunk, copying the matching updates out into caller-owned staging. Extraction is finished with a chunk's body when the call returns, so a spilled body is read for the scope of one call and no reference into pool memory exists outside it -- the contract that lets a buffer pool evict with no reader accounting. The trait has no key/val/time/diff opinions (Staging and Probes are family-chosen), and the one comparison the batch driver needs is delegated to the chunk via locate, answerable from resident metadata. Chunk boundaries are carried as the consume-index protocol: a chunk consumes every probe strictly below its last key and extracts-but-does-not-consume one equal to it, re-offered to the next chunk as a legal straddle. Ported from the differential fork (TimelyDataflow/differential-dataflow PR 782) so consumers here depend on a local definition rather than a git pin. One delta from the upstream shape: the batch driver is the UnloadBatch extension trait rather than inherent methods on ChunkBatch (not available to this crate); names and signatures match, so call sites survive a future switch to upstream re-exports unchanged. If the PR lands, this module retires in favor of re-exports. Contract test: a miniature row family driven through every (chunk cut x contiguous probe range) placement with a multi-chunk-spanning key, against the reference filter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Part of the buffer-managed dataflow state work (
doc/developer/design/20260610_buffer_managed_state.md, pool in #37718, configuration in #37719). This is the read capability the pool-spillable chunk stack builds on: the ColumnChunk PR stacks directly on this one, and upsert v2's stash drain is the production consumer. Upstream counterpart: TimelyDataflow/differential-dataflow#782.Part of CPU-184.
Description
UnloadChunk is a second way to read a chunk, alongside the cursor: hand in a sorted, deduplicated column of probe keys and the matching updates are appended to caller-owned staging.
locateanswers where a probe falls relative to a chunk's key span from resident metadata only, so the batch driver can gallop the chunk list and open only chunks whose span contains a probe. Probes that fall in a gap between chunks are consumed without faulting any body. A probe equal to a chunk's last key is extracted but re-offered to the next chunk, since its group may continue there (the straddle protocol).This PR is the trait and the batch-level driver only. The ColumnChunk implementation, including the spilled-body read path, is the next PR in the stack.
Verification
Contract tests over a miniature row family drive extraction across arbitrary chunk cuts and probe placements, straddled keys included, and compare against a reference filter of the raw rows.