Make shared string builders fallible end-to-end with try_* APIs#23223
Open
kosiew wants to merge 10 commits into
Open
Make shared string builders fallible end-to-end with try_* APIs#23223kosiew wants to merge 10 commits into
try_* APIs#23223kosiew wants to merge 10 commits into
Conversation
- Enhanced BulkNullStringArrayBuilder to implement a fallible `try_*` contract. - Introduced compatibility wrappers for infallible methods. - Added new methods to StringViewArrayBuilder: - `try_append_byte_map` - `try_append_with` - Implemented fallible long-view part validation and error capture with rollback for the writer. - Included test-only failing bulk builder and added success + overflow tests to ensure robustness.
- Reused actual conversion error in `write_str` and `write_char` - Simplified rollback error path in `try_append_with` - Moved failing test helper types into the tests module - Deduplicated failing test error via `failing_overflow()`
…for improved test reuse - Moved FailingStringWriter and FailingBulkNullStringArrayBuilder to allow for downstream crate module tests to reuse them via `crate::strings::FailingBulkNullStringArrayBuilder`. - Updated visibility to `#[cfg(test)] pub(crate)` in `datafusion/functions/src/strings.rs`.
…in StringViewArrayBuilder
…as pub(crate) for test reuse
- Removed `append_byte_map` and `append_with` from `GenericStringArrayBuilder` - Removed `append_byte_map` and `append_with` from `StringViewArrayBuilder` - Retained trait default methods on `BulkNullStringArrayBuilder` - Updated documentation to fix broken `Self::...` links
kosiew
marked this pull request as ready for review
June 28, 2026 13:40
Contributor
Author
Contributor
|
Have we ever seen StringView overflow errors? I think to get them we would need to have individual Strings that are larger than 2GB (overflow an i32) I thought a far more common issue was to overflow a String/Utf8 array (where the total length of all strings in a single Array can't be over 2GB) Basically I wonder if it is worth the extra code complexity for a case that doesn't really happen |
Contributor
|
My concern with a refactor like this is that it doesn't slow down performance of the string functions. Can we run the relevant string function benchmarks? |
alamb
reviewed
Jul 7, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thank you for this PR @kosiew -- I took a brief look and the basic idea looks good (remove the infallable methods) but it seems pretty complicated to me
I would like to know:
- Does this change have any performance implications (aka let's run some benchmarks)
- Can we make it simpler (or maybe break the PR down into smaller parts)? The delayed error handling for StringWriter seems to be one of the more complicated parts
Perhaps @Omega359 or @neilconway can help review this as I think they wrote a good portion of the initial code
| } | ||
|
|
||
| fn try_string_view_part(field: &str, value: usize) -> Result<u32> { | ||
| // StringView stores these fields as u32, but Arrow limits lengths, |
Contributor
There was a problem hiding this comment.
I think it is unlikely buffer indices will ever overflow a i32 -- that would mean there are 2 billion buffers (not values)
However, I see this is already checked below
| /// `None` while the row fits inline; becomes `Some(start)` (offset of | ||
| /// the row's first byte in `in_progress`) at first spill. | ||
| spill_cursor: Option<usize>, | ||
| error: Option<DataFusionError>, |
Contributor
There was a problem hiding this comment.
why does this need delayed error handling? It seems like the code would be a lot simpler if the error was returned immediately
If we do need delayed reporting, can we please add a comment about why ?
Contributor
Author
There was a problem hiding this comment.
The delayed error is needed because the existing StringWriter API has infallible write_str / write_char methods, and try_append_with accepts a closure using that writer.
Once a write overflows, the writer has no Result channel to return through immediately, so it records the first error and try_append_with returns it after the closure finishes, rolling back in_progress to old_len.
Added comment.
| let offset = try_string_view_part("offset", start)?; | ||
| let buffer_index = | ||
| try_string_view_part("buffer count", self.completed.len())?; | ||
| self.views.push(Self::make_long_view_checked( |
Contributor
There was a problem hiding this comment.
It is strange to me that this is called make_long_view_checked but doesn't seem to return an error
Contributor
Author
There was a problem hiding this comment.
Renamed it to make_long_view_from_checked_parts
| } | ||
|
|
||
| impl StringViewWriter<'_> { | ||
| #[inline] |
Contributor
There was a problem hiding this comment.
can we please document in comments what this method is doing (likewise for write_spilled_bytes)?
…nd and add BulkNullStringArrayBuilder override - Restored the fast infallible `GenericStringArrayBuilder::append_with` function. - Added `BulkNullStringArrayBuilder` override to allow generic hot calls to avoid the `try_append_with` wrapper. - Kept the existing `try_append_with` method for fallible and rollback-safe operations.
- Removed restored panic fast path in GenericStringArrayBuilder::append_with - Eliminated trait override that leveraged the removed panic fast path - Optimized try_append_with: - Removed rollback branch - Validates offset using try_offset - Returns DataFusionError - Documentation updates to advise discarding builder after an error
Contributor
|
I'll try and take a look at this tomorrow. |
kosiew
marked this pull request as draft
July 8, 2026 13:37
Contributor
Author
|
benchmark results from cargo bench -p datafusion-functions --bench upper cargo bench -p datafusion-functions --bench reverse cargo bench -p datafusion-functions --bench translate cargo bench -p datafusion-functions --bench replace cargo bench -p datafusion-functions --bench concat cargo bench -p datafusion-functions --bench concat_ws |
Contributor
|
│ array_from_to │ 0.02 / 0.02 ±0.00 / 0.02 ms │ 0.02 / 0.02 ±0.00 / 0.03 ms │ 1.27x slower │ hmm. |
- Clarified `try_string_view_part` buffer-index guard - Enhanced documentation for `StringView` overflow - Renamed `make_long_view_checked` to `make_long_view_from_checked_parts` - Added comments for delayed `StringViewWriter` error handling - Documented `fail` method - Documented `write_spilled_bytes` method
Contributor
Author
I agree these StringView overflows are much less common than normal Utf8 offset overflow. For StringView, the practical cases are huge individual/incrementally-written values or invalid long-view parts; total array bytes can be split across buffers, so this is not the common “total Utf8 array >2GB” case. I think keeping fallible StringView paths is reasonable because the shared |
kosiew
marked this pull request as ready for review
July 8, 2026 14:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This change moves the shared string builder infrastructure to fallible
try_*APIs so overflow conditions can be propagated asDataFusionErrorinstead of requiring panic-based handling. This provides a consistent error propagation path for downstream UDF migrations while preserving the existing infallible APIs as compatibility wrappers where appropriate.This PR is preparatory work for migrating downstream string UDFs onto fallible append/write APIs end-to-end. The follow-up work will cover direct row emitters such as
chr,uuid,initcap, andsubstr; helper-driven writers such asoverlay,reverse, andtranslate; the largersplit_partmigration, including its index-normalization helpers; and output-amplifying functions such asrepeat,lpad, andrpad, where oversized output must returnDataFusionErrorrather than panic.What changes are included in this PR?
Add shared helpers for validating
StringViewlength, offset, and buffer index values against Arrow'si32::MAXlimits.Introduce fallible
try_*methods throughoutBulkNullStringArrayBuilder:try_append_valuetry_append_placeholdertry_append_withtry_append_byte_mapKeep the existing infallible
append_*methods as compatibility shims that delegate to the correspondingtry_*methods and panic on overflow.Convert
GenericStringArrayBuilder::append_withto reuse the fallible implementation instead of duplicating logic.Refactor
StringViewArrayBuilderto:try_append_withandtry_append_byte_map,Add shared test-only utilities (
FailingBulkNullStringArrayBuilderandFailingStringWriter) to support overflow propagation tests in this and downstream modules.Prepare shared string builder APIs for follow-up UDF migrations covering direct append call sites, helper-driven row writers,
split_partindex handling, and output-amplifying functions such asrepeat,lpad, andrpad.Are these changes tested?
Yes.
This PR adds the following tests:
bulk_try_append_methodsstring_view_builder_try_append_with_and_byte_map_success_pathstring_view_builder_rejects_long_view_part_overflowfailing_bulk_builder_propagates_try_append_errorsIt also continues to exercise existing string builder tests.
Are there any user-facing changes?
No user-facing behavior is intended. This is shared internal infrastructure that enables downstream code to propagate overflow errors through fallible APIs while preserving the existing infallible compatibility methods.
LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.