fix(blob): compact struct-nested blob v2 columns - #8083
Draft
wjones127 wants to merge 1 commit into
Draft
Conversation
Compaction converted blob v2 descriptor columns back into the writer's user view only for top-level columns. A blob v2 field nested inside a struct was passed through as a descriptor struct, so any fragment rewrite failed with "Blob struct missing `data` field" and the dataset could never be compacted. Walk struct children when rewriting a batch (and when building the transformed stream schema), addressing each blob leaf by its dotted path. Blob v2 under a list is still unsupported, but now reports NotSupported instead of the misleading descriptor error. Fixes lance-format#8082 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compaction rewrites blob v2 columns from their stored descriptor form back into the writer's user view, but it only did so for columns at the top level of the schema. A blob v2 field nested inside a struct — e.g.
struct<mime: string, payload: blob>— was passed through untouched, so the writer received a descriptor struct where it expected a payload and every fragment rewrite failed with "Blob struct missingdatafield". The dataset was still readable, but permanently un-compactable: it could never reclaim space from deleted rows or merge small fragments.This PR walks struct children when rewriting a batch, and when building the transformed stream schema, addressing each blob leaf by its dotted path so its payloads are read back correctly.
Fixes #8082
Not included
Blob v2 nested under a list is still not supported by compaction: payloads are addressed one per row, and a list holds many values per row. That case previously hit the same misleading "missing
datafield" error and now reportsNotSupportedwith an explanation instead.