Add preimage optimization for ceil to rewrite predicates into range filters#20541
Open
kosiew wants to merge 7 commits intoapache:mainfrom
Open
Add preimage optimization for ceil to rewrite predicates into range filters#20541kosiew wants to merge 7 commits intoapache:mainfrom
ceil to rewrite predicates into range filters#20541kosiew wants to merge 7 commits intoapache:mainfrom
Conversation
…ng SQL logic tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
ceilpart of Implementpreimageforceilandroundfunctions #20197.Rationale for this change
DataFusion’s preimage framework can turn predicates on deterministic functions into equivalent predicates on the underlying column(s). For
ceil(x), the mathematical preimage for a target integer valueNis the interval (N − 1, N].Without a preimage implementation, filters such as
WHERE ceil(col) = 6must evaluateceilfor every row, which can inhibit predicate pushdown and other optimizer wins.This PR implements
ceil’s preimage to enable rewriting comparisons into simple range predicates (with careful handling of floating-point representation boundaries and decimals), improving the optimizer’s ability to push filters down to scans and reduce work during execution.What changes are included in this PR?
Implemented
ScalarUDFImpl::preimagefor theceilscalar function.ceil(x) = Nas a half-open interval suitable for theIntervalframework.next_upfor floating-point bounds so that the strict lower bound(N-1, …]is represented safely asx >= next_up(N-1)and the inclusive upper bound… <= Nbecomesx < next_up(N).N - 1collapses toNdue to float spacing (e.g., above2^53forf64, above2^24forf32).Added decimal preimage support for
Decimal32/64/128/256.10^-scale) to represent(N-1, N]as[N-1+step, N+step).[N, N+1).Added unit tests covering:
PreimageResult::None.Added a new SQLLogicTest file
ceil_preimage.slt.EXPLAINfor=,IN,IS [NOT] DISTINCT FROM, and boundary cases.Are these changes tested?
Yes.
Rust unit tests added in
datafusion/functions/src/math/ceil.rsvalidate:SQLLogicTest added in
datafusion/sqllogictest/test_files/ceil_preimage.sltvalidates:EXPLAIN(including floatnext_upbounds and decimal bounds).Are there any user-facing changes?
No user-visible behavior changes are intended. The semantics of
ceilare unchanged.This is an optimizer improvement that may:
ceil.ceil(col)by enabling range filtering and better predicate pushdown.No documentation updates are required.
LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.