-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Support NULL time in First/Last/FirstBy/LastBy aggregations #17064
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request adds comprehensive support for NULL timestamps in FIRST, LAST, FIRST_BY, and LAST_BY aggregation functions. The implementation prioritizes data with valid timestamps over NULL timestamp data, and when only NULL timestamp data exists, adopts a "First Null Wins" strategy where the first encountered NULL time value is retained.
Changes:
- Added
initNullTimeValuestracking field to all grouped accumulators andinitNullTimeValueto non-grouped accumulators - Enhanced serialization format to include an additional boolean flag indicating whether the order time is NULL
- Implemented separate update methods for NULL time values across all data types (INT32, INT64, FLOAT, DOUBLE, BINARY, BOOLEAN)
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| GroupedLastByAccumulator.java | Added NULL time tracking with initNullTimeValues field and separate update paths for NULL vs valid timestamps |
| GroupedLastAccumulator.java | Similar NULL time support for LAST aggregation with dual initialization tracking |
| GroupedFirstByAccumulator.java | Mirrored LAST logic but with minimum time comparison for FIRST semantics |
| GroupedFirstAccumulator.java | NULL time handling for non-BY FIRST aggregation |
| Utils.java | New serialization methods serializeTimeValueWithNull to handle the extended format with NULL time flags |
| LastDescAccumulator.java | Descending order variant with NULL time value handling |
| LastByDescAccumulator.java | Descending LAST_BY with NULL time support |
| LastByAccumulator.java | Non-grouped LAST_BY with NULL time tracking |
| LastAccumulator.java | Non-grouped LAST with NULL time support |
| FirstDescAccumulator.java | Descending FIRST with NULL time handling |
| FirstByDescAccumulator.java | Descending FIRST_BY with NULL time support |
| FirstByAccumulator.java | Non-grouped FIRST_BY with NULL time tracking |
| FirstAccumulator.java | Non-grouped FIRST with comprehensive NULL time support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...b/db/queryengine/execution/operator/source/relational/aggregation/LastByDescAccumulator.java
Outdated
Show resolved
Hide resolved
...b/db/queryengine/execution/operator/source/relational/aggregation/LastByDescAccumulator.java
Outdated
Show resolved
Hide resolved
...iotdb/db/queryengine/execution/operator/source/relational/aggregation/LastByAccumulator.java
Outdated
Show resolved
Hide resolved
...iotdb/db/queryengine/execution/operator/source/relational/aggregation/LastByAccumulator.java
Show resolved
Hide resolved
.../db/queryengine/execution/operator/source/relational/aggregation/FirstByDescAccumulator.java
Outdated
Show resolved
Hide resolved
...otdb/db/queryengine/execution/operator/source/relational/aggregation/FirstByAccumulator.java
Outdated
Show resolved
Hide resolved
...ngine/execution/operator/source/relational/aggregation/grouped/GroupedLastByAccumulator.java
Outdated
Show resolved
Hide resolved
...ngine/execution/operator/source/relational/aggregation/grouped/GroupedLastByAccumulator.java
Outdated
Show resolved
Hide resolved
.../org/apache/iotdb/db/queryengine/execution/operator/source/relational/aggregation/Utils.java
Outdated
Show resolved
Hide resolved
7b11bf8 to
6ee894a
Compare
6ee894a to
ab5757f
Compare
Description:
This PR introduces robust support for NULL timestamps in FIRST, LAST, FIRST_BY, and LAST_BY aggregation functions. Previously, rows with NULL time might have been handled inconsistently or ignored.
Key Changes:
Null Time Handling Strategy:
Priority: Data with a Valid Time always takes precedence over data with a Null Time.
Fallback Policy: If only NULL time data exists, the aggregations now adopt a "First Null Wins" strategy. This means the first encountered value with a NULL timestamp is retained as the result, ensuring consistent behavior across both First and Last families.
Comprehensive Support:
Applied these changes to both standard Accumulators (for global aggregation) and GroupedAccumulators (for GROUP BY queries).
Covered all supported data types (INT32, INT64, FLOAT, DOUBLE, BINARY, BOOLEAN).