Skip to content

clone partial columns of aligned tvlist for query - #18409

Open
shuwenwei wants to merge 8 commits into
masterfrom
partial_tvlist_clone
Open

clone partial columns of aligned tvlist for query#18409
shuwenwei wants to merge 8 commits into
masterfrom
partial_tvlist_clone

Conversation

@shuwenwei

Copy link
Copy Markdown
Member

Cherry-picked from #18394 ([To dev/1.3] clone partial columns of aligned tvlist for query).

Commit: edeb7c5276

For queries that touch only a few columns of a wide aligned device, clone only the accessed columns of the working AlignedTVList instead of the whole list, then swap the working list so the unaccessed columns are moved out. This dramatically reduces the memory retained by such queries on wide aligned devices.

This cherry-pick also carries over the follow-up fixes from the dev/1.3 PR:

  • prepareTvListMapForQuery retries under the memChunk lock after acquiring the candidate's queryListLock, so a swapped working list can never be operated on while holding a stale lock.
  • Memory reservation is made before clone allocation and rolled back on failure; the clone and working-list swap happen atomically under the memChunk critical section.
  • Non-query columns are released before ownership transfer to the first query.
  • Aligned RamInfo accounting keeps reservation and cleanup sizes consistent.

Adapted to the current master APIs (IFullPath/AlignedFullPath, ModEntry, table-model chunk metadata, i18n messages) during conflict resolution.

@shuwenwei shuwenwei changed the title [To master] clone partial columns of aligned tvlist for query clone partial columns of aligned tvlist for query Aug 6, 2026
…ager test fakes

The cherry-pick added the releaseMemoryImmediately(long) method to the
calc-commons MemoryReservationManager interface, but four test classes that
implement the interface directly were not updated, breaking test compilation.
Add the missing method to each fake, mirroring the semantics of the existing
releaseMemoryCumulatively implementation in each class.
- ResourceByPathUtilsTest: getResourceInstance(IFullPath) requires a
  NonAlignedFullPath; MeasurementPath does not implement IFullPath, so the
  test no longer compiled. Build the NonAlignedFullPath from
  IDeviceID.Factory.DEFAULT_FACTORY and a MeasurementSchema.
- FragmentInstanceExecutionTest: use IDeviceID.Factory.DEFAULT_FACTORY.create
  directly instead of DeviceIDFactory.getInstance().getDeviceID(new
  PartialPath(...)), matching the query-side NonAlignedFullPath construction
  in the same file, and drop the now-unused imports.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.47748% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.64%. Comparing base (7e2c453) to head (6be2c09).
⚠️ Report is 8 commits behind head on master.

Files with missing lines Patch % Lines
...engine/schemaregion/utils/ResourceByPathUtils.java 66.00% 34 Missing ⚠️
...he/iotdb/db/utils/datastructure/AlignedTVList.java 90.15% 19 Missing ⚠️
...ne/execution/fragment/FragmentInstanceContext.java 16.66% 15 Missing ⚠️
...ner/memory/ThreadSafeMemoryReservationManager.java 0.00% 2 Missing ⚠️
...e/dataregion/memtable/AlignedReadOnlyMemChunk.java 0.00% 2 Missing ⚠️
.../planner/memory/FakedMemoryReservationManager.java 0.00% 1 Missing ⚠️
.../dataregion/memtable/AbstractWritableMemChunk.java 83.33% 1 Missing ⚠️
...geengine/dataregion/memtable/ReadOnlyMemChunk.java 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18409      +/-   ##
============================================
+ Coverage     43.61%   43.64%   +0.02%     
  Complexity      374      374              
============================================
  Files          5393     5399       +6     
  Lines        385958   386593     +635     
  Branches      50267    50401     +134     
============================================
+ Hits         168337   168712     +375     
- Misses       217621   217881     +260     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shuwenwei
shuwenwei marked this pull request as ready for review August 10, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant