Improve ZIP store performance and filesystem listing#70
Open
kulvait wants to merge 7 commits intozarr-developers:mainfrom
Open
Improve ZIP store performance and filesystem listing#70kulvait wants to merge 7 commits intozarr-developers:mainfrom
kulvait wants to merge 7 commits intozarr-developers:mainfrom
Conversation
- Replace stream-based ZIP access with random-access `ZipFile` for faster and more efficient reads - Optimize ZIP store initialization by reading the ZIP index only, avoiding full stream traversal - Introduce improved caching of directory structure and file sizes using synchronized maps - Simplify internal logic and remove unnecessary dependencies - Improve filesystem listing performance Details: - Removed dependency on Apache Commons ZipArchiveInputStream and related classes in ReadOnlyZipStore - Added efficient caching using maps for directories and file sizes, async friendly - Normalized entry names for consistent lookup - Reduced redundant computations (e.g. cached entry sizes) - Simplified stream handling and chunk calculation These changes significantly improve performance, reduce complexity, and make the ZIP store implementation more maintainable.
… and directory existence
- Problem:
* Some tests failed because ReadOnlyZipStore could not locate ZIP entries when the store
was created by simply zipping a directory. Tools differ: some produce entry names with
a leading slash, others without. This caused getInputStream(), read(), and getSize() to return null.
* testExists() failed for root keys because exists() included directories, but by design it should only test file existence.
- Solution:
* Added resolvePathWithLeadingSlashFromKeys() to try a secondary lookup with a leading slash.
Primary lookup uses the standard key without leading slash; secondary is only for compatibility.
* Modified exists(String[] keys) to check only fileSizeIndex, ignoring directories, to match the intended design.
- Effect:
* ReadOnlyZipStoreTest passes regardless of whether ZIP entries have leading slashes or not.
* Test logic now clearly distinguishes between file existence and directory entries.
Author
|
I have updated the PR so it fixes failed workflow and tests. Can you please run it again? |
This reverts a previous attempt to simplify the FilesystemStore#list
implementation using a parallel stream and String-based path splitting:
Files.walk(rootPath)
.filter(Files::isRegularFile)
.parallel()
.map(path -> rootPath.relativize(path)
.toString()
.split(File.separator));
While this approach showed performance improvements in some scenarios
(likely due to .parallel() decoupling filesystem traversal from mapping),
it introduced a platform-specific issue: String.split() expects a regex,
and File.separator ("\" on Windows) leads to a PatternSyntaxException.
A correct fix would require quoting the separator, e.g.:
split((java.util.regex.Pattern.quote(File.separator))
However, this change is outside the scope of the current PR, which focuses
on ZipStore improvements rather than altering core FilesystemStore behavior.
Revert to the upstream commit c9a5ee1 for correctness and consistency.
As a follow-up, it may be worth evaluating the use of .parallel() separately,
as it appears to provide measurable benefits in some workloads.
Change of ReadOnlyZipStore.java is formatting only
Author
|
Due to remaining Windows test failures, I reverted FilesystemStore to the upstream implementation, see commit message. This change should restore cross-platform compatibility and allow all tests to pass. Please approve workflow. |
…resource leaks Both get(...) and getInputStream(...) now delegate to a shared helper method that reads ZIP entry data into a byte[]. This refactor removes duplicated ZIP handling logic (entry lookup, range validation, skip/read loop) and ensures consistent behavior across both APIs. Previous implementation of getInputStream() returned an InputStream backed by a ZipFile entry. If callers failed to close the stream, as is a case in one of the tests in StoreTest.java, on Windows this caused file locking issues, wchich prevented deletion of the underlying ZIP file during tests.
- Clean up debugging and temporary logging statements - Rename ensureCacheNew to buildZipIndex for clarity - Replace synchronized HashSet wrappers with ConcurrentHashMap.newKeySet() to improve performance and reduce locking overhead during indexing
Author
|
I hope this PR will be considered for merge, as I put a lot of effort into addressing issues that were triggering test failures in CI. The changes also meaningfully improve the performance and efficiency of the ReadOnlyZipStore implementation. |
normanrz
reviewed
Apr 24, 2026
Member
normanrz
left a comment
normanrz
left a comment
There was a problem hiding this comment.
Thanks for the PR!
I think this should be a new class, because this implementation with ZipFile only works on the filesystem (FilesystemStore), whereas the existing ReadOnlyZipStore also works with remote stores (e.g. HttpStore, S3Store).
Thank you for the review, these suggestions integrate cleanly. I will review remaining suggestions and comment or integrate them. Co-authored-by: Norman Rzepka <code@normanrz.com>
Renamed normalizeEntryName to stripLeadingAndTrailingSlashes for better clarity. Enhanced the normalization logic to handle multiple consecutive leading or trailing slashes. Now, any degenerate entries with multiple slashes are properly stripped to ensure consistent and safe entry paths. Removed redundant checks for leading or trailing slashes after normalization. -> Based on Norman Rzepka's suggestion. Since the normalization process now handles all cases, there's no need for further slash-stripping. Updated zip entry processing logic to: Continue using the isDirectory() flag and directoryToChildrenDirectoriesIndex for directory entries. Unified root directory handling, creating indices, no recursion.
Author
|
Thank you, Norman, for your comments and suggestions! Personally, I don't have a strong preference on whether this should be implemented as a new class or not. From my perspective, in my plugin, reading the ZIP-backed Zarr is becoming more responsible than just the on-disk representation. It would be nice to support e.g. S3Store via HttpRange requests, but at the moment, this would be lot of work. Feel free to rename the class. |
normanrz
reviewed
May 4, 2026
Member
normanrz
left a comment
normanrz
left a comment
There was a problem hiding this comment.
Thanks for the edits. This needs to be a new class with clear instructions for the users that it only works for filesystem stores. Maybe call it ReadOnlyFilesystemStore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation for this work was that on large zip stores, the indexing was very slow and then opening arrays as well.
So I did replaced Apache Commons ZipArchiveInputStream by random access java class.
Change in FilesystemStore.java is more cosmetic but also slightly increase performance.
ZipFilefor faster and more efficient readsDetails:
These changes significantly improve performance, reduce complexity, and make the ZIP store implementation more maintainable.