Stabilize zarr.core.sync.sync() for downstream async integration

[aladinor]

Context

We're adding async support to xarray's open_datatree (xarray PR #10742) to enable concurrent I/O when opening multi-group zarr stores. This achieves at least ~6x speedup on cloud data (8.5s → 1.4s) by opening groups concurrently using zarr's async APIs.

During review, @shoyer raised concerns about xarray relying on zarr internal APIs that could break without notice.

What xarray uses from zarr

API	Public?	Usage
zarr.AsyncGroup	Yes (in __all__)	Type checking, async group operations
AsyncGroup.members()	Yes (documented method)	Listing child groups asynchronously
AsyncGroup.open()	Yes (documented classmethod)	Opening groups asynchronously
zarr.core.sync.sync()	No (not re-exported, not in __all__)	Bridging async→sync in xarray's open_datatree()

Why we need zarr.core.sync.sync()

xarray's open_datatree() is synchronous, but we want to use zarr's AsyncGroup APIs internally to open multiple groups concurrently. We use zarr.core.sync.sync() to bridge async→sync — the same function zarr uses internally via SyncMixin._sync() to implement its sync Group class.

Without a supported bridge, xarray would need to manage its own event loop, which is error-prone and could conflict with zarr's internal loop management.

Example usage in xarray:

from zarr.core.sync import sync as zarr_sync

# In xarray's sync open_datatree path
results = zarr_sync(create_indexes_async())

The ask

1. Could zarr.core.sync.sync() be made public? It would just need to be re-exported in zarr/__init__.py and added to __all__. This seems low-cost since zarr already depends on it extensively internally.

2. Or is there an alternative supported mechanism for downstream libraries to run async zarr operations from sync contexts?

3. Can we also confirm that AsyncGroup.members() return type (AsyncGenerator[tuple[str, AsyncArray | AsyncGroup], None]) is considered stable?

Alternatives we've considered

• xarray manages its own event loop — risky, likely conflicts with zarr's loop
• Higher-level zarr API (e.g., Group.open_members() returning all child groups concurrently) — would eliminate the need for xarray to manage concurrency, but more work on zarr's side

References

• xarray PR: Implement async support for open_datatree pydata/xarray#10742
• xarray async strategy discussion: How should Xarray control asynchronous calls? pydata/xarray#10622

cc @TomNicholas @shoyer @keewis @dcherian @ilan-gold @d-v-b

We'd love to hear your thoughts, suggestions, or alternative approaches. Any feedback from the zarr core team on the best path forward would be greatly appreciated!

[aladinor]

Also worth noting: @BorisTheBrave has proposed the same approach in xarray PR #11171 — using zarr's sync() to share zarr's internal event loop so the sync API gets async benefits transparently. See his comment: pydata/xarray#10622 (comment)

This reinforces the case for making zarr.core.sync.sync() a stable, public API — multiple downstream efforts are converging on it as the right bridge between sync and async.

[shoyer]

Another (non-exclusive) idea worth considering is forking Zarr's Jupyter-safe event loop logic into a separate library, which could be used by other projects like Xarray. I think we would still need a way to to invoke Zarr's event loop from inside Xarray for async Zarr IO, though.

[d-v-b]

do you need to interface with existing async xarray routines, or do you just need to synchronously invoke something that should be done asynchronously under the hood?

If it's the second option, shouldn't we add an API to zarr-python that does this? Like, you say you need to open multiple groups concurrently. That means reading N metadata documents concurrently. that means using the async _get_many method for stores, then wrapping that in blocking code, and making it public. We already have the async method, but we don't have a public API for it.

More broadly, we don't have any public synchronous API for our store classes, which IMO is a major oversight, because this would evidently be useful to people. I tried to move in this direction with this pr: #3638, but the result was private methods. IMO we should consider a public, synchronous store API, so people can do useful concurrency-backed store operations without needing to touch our event loop.

[BorisTheBrave]

Sorry, I'm not referring to that.

I mean methods like create_hierarachy. I recall seeing discussion on the xarray side, that it would be preferable to use this method (because it does as concurrently what would otherwise be many separate calls). I fact, i think it may have been added to zarr on the request of xarray? But because of it's batched nature, it was difficult to fit into the existing code. And the code would still not be complete free from sequentiality - why wait for all metadata to be written before you start to write array data?

open_members is probably similar. I haven't looked at it, but is it doing significantly more than zarr.sync(gather(...))?

[d-v-b]

I am confused, if you trying to add a new async API to xarray, then you don't need sync, because you would use plain async functions and await them. And if you looking to concurrently perform Zarr IO from synchronous code, which is where zarr-python using sync today, then why exactly does xarray need access to sync, vs access to a sync API that uses it internally?

[d-v-b]

why wait for all metadata to be written before you start to write array data?

that's a valid point. You can do this today with zarr-python but its kind of obscure, we should make this easier to do.

[BorisTheBrave]

I'm an interested party, not an xarray developer so I cannot answer with certainty, but...

if you trying to add a new async API to xarray, then you don't need sync, because you would use plain async functions and await them.

The focus of their discussions is making the existing sync API faster by being more concurrent under the hood, not adding an async layer.

And if you looking to concurrently perform Zarr IO from synchronous code, which is where zarr-python using sync today, then why exactly does xarray need access to sync, vs access to a sync API that uses it internally?

How do you see xarray calling multiple methods of the zarr sync API concurrently? It would need a threadpool of its own, clearly a poor solution.

[d-v-b]

you don't need a threadpool for concurrency. It seems to me that zarr python needs a synchronous routine that takes a representation of the stuff xarray wants to do (create groups, create arrays, write chunks), and runs those actions concurrently. very schematically:

def do_zarr_stuff(metadata_to_create: Mapping[str, metadata_object], chunks_to_write: Mapping[str, NDArray]) -> ResultObject:
    # concurrently create metadata and write chunks
    # return ResultObject when done

we could also make this an iterable that yields stuff as its completed. the point is that this is a synchronous routine, defined in zarr-python, that xarray uses. I don't think xarray's usage of zarr is unique. we should design a solution that works for all zarr-python users, not just xarray.

[ilan-gold]

Not to pollute the conversation, but along the lines of what @shoyer said, just an idea - what would the implications be of allowing applications to provide their own event loops for zarr-python to use?

[d-v-b]

seems like a direction that anyio could help with: see https://www.prefect.io/blog/oss-love-letters-how-anyio-powers-prefects-async-architecture.

IMO this direction makes sense for another async library that wants to work with zarr-python. I'm not sure that describes the xarray situation, though