fix(ArraySpec): proper and robust equality semantics for ArraySpec by checking fill_value for byte-identicality, fixes #3054. by sehoffmann · Pull Request #4183 · zarr-developers/zarr-python

sehoffmann · 2026-07-23T12:06:59Z

Summary

ArraySpec describes the exact layout for an array. The previously auto-generated __eq__ and __hash__ methods from @dataclass did not reflect this correctly due to the fill_value attribute.

Specifically:

For IEEE float nans: nan != nan and thus ArraySpec(**kwargs) != ArraySpec(**kwargs) if the fill value is nan (although it they are in fact identical as they have the same fill).
Same issue if numpy's np.datetime64('NaT') which is also not self-equal
On the other hand, float(-0.0) == float(0.0) despite these two fill_value's being different float numbers. They would result in different arrays in memory and on disk.
Most importantly, if fill_value is a np.void, such as from a structured dtype, hash(ArraySpec(...)) failed outright, because (mutable) np.voids are not hashable. (Issue sharding codec use of lru caching fails with numpy void scalars #3054).

This PR fixes all of the above by comparing fill_value using byte-identity, which should be the correct semantics for a fill value. It also adds a comprehensive test suite for the equality and hash semantics of ArraySpec which was non-existent before.

Before this fix/PR, the following (newly added) tests failed:

FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_hashable[structured-void]** - TypeError: unhashable type: 'writeable void-scalar'
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_equal_specs_hash_equal[float64-nan]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Float64(endianness='little'), fill_value=np.float64(nan), config=ArrayConfig(order='C', ...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_equal_specs_hash_equal[datetime64-NaT]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=DateTime64(endianness='little', scale_factor=1, unit='s'), fill_value=np.datetime64('NaT...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_equal_specs_hash_equal[structured-void]** - TypeError: unhashable type: 'writeable void-scalar'
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_distinct_specs_unequal[shape-structured-void]** - TypeError: unhashable type: 'writeable void-scalar'
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_distinct_specs_unequal[order-structured-void]** - TypeError: unhashable type: 'writeable void-scalar'
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_distinct_specs_unequal[prototype-structured-void]** - TypeError: unhashable type: 'writeable void-scalar'
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[float16]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Float16(endianness='little'), fill_value=np.float16(-0.0), config=ArrayConfig(order='C',...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[float32]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Float32(endianness='little'), fill_value=np.float32(-0.0), config=ArrayConfig(order='C',...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[float64]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Float64(endianness='little'), fill_value=np.float64(-0.0), config=ArrayConfig(order='C',...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[complex128-both]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Complex128(endianness='little'), fill_value=np.complex128(-0-0j), config=ArrayConfig(ord...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[complex128-imag]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Complex128(endianness='little'), fill_value=np.complex128(-0j), config=ArrayConfig(order...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[complex128-real]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Complex128(endianness='little'), fill_value=np.complex128(-0+0j), config=ArrayConfig(ord...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...
FAILED tests/test_array_spec.py::**TestArraySpecHashEq::test_signed_zero_fills_are_distinct[structured]** - AssertionError: assert ArraySpec(shape=(4, 4), dtype=Struct(fields=(('a', Float64(endianness='little')),)), fill_value=np.void((-0.0,), dtype...otype=BufferPrototype(buffer=<class 'zarr.core.buffer.cpu.B...

With the fix applied, all tests pass. Also, before this fix, the following valid zarr code would throw an exception:

t = np.dtype([("a", "i4"), ("b", "f4")])

arr = zarr.create_array(
    MemoryStore(),
    shape=(8,),
    chunks=(2,),
    shards=(4,),
    dtype=dt,
    fill_value=(0, 0.0),
)
arr[:] = np.ones(8, dtype=dt)  # -> TypeError: unhashable type: 'writeable void-scalar'

This PR fixes the above.

Behavioral Changes (summarized):

Structured / np.void fills: hash(spec) no longer raises TypeError; such specs are now hashable and usable as dict/set/cache keys. (Fixes sharding codec use of lru caching fails with numpy void scalars #3054.)
nan fills: two specs with a NaN fill now compare equal and hash equal; before they compared unequal (nan != nan).
NaT fills: same change as NaN: now equal, were unequal.
Signed zero (+0.0 vs -0.0): now compare unequal (distinct fills, distinct bytes); before they compared equal.
Comparison basis: numpy-scalar fills are now compared by exact bit pattern rather than numeric ==, so equality no longer inherits numpy's value-promotion quirks. (e.g. see BUG: numpy int64/float64 comparisons are inexact numpy/numpy#32071)

For reviewers

#3054 discusses getting rid of the lru cache entirely. This is an orthogonal design decision in my opinion. While this PR fixes the exception, it does not remove the lru cache. Instead, it ensures that ArraySpec has robust hash semantics in the first place which is a good thing regardless of #3054 and whether an lru cache is used or not.

Author attestation

I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

TODO

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.md
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

… checking fill_value for byte-identicality, fixes zarr-developers#3054.

codecov · 2026-07-23T12:16:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.85%. Comparing base (80e00ae) to head (55f096e).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4183   +/-   ##
=======================================
  Coverage   93.84%   93.85%           
=======================================
  Files          91       91           
  Lines       12549    12561   +12     
=======================================
+ Hits        11777    11789   +12     
  Misses        772      772

Files with missing lines	Coverage Δ
src/zarr/core/array_spec.py	`100.00% <100.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

sehoffmann · 2026-07-23T12:18:18Z

Should I add a test case that non-ArraySpec comparison raises as requested by the codecov plugin? IMO this is a trivial case..

d-v-b · 2026-07-23T19:25:11Z

Thanks, this is a great fix. I will leave a few comments, but IMO this is good to go

Should I add a test case that non-ArraySpec comparison raises as requested by the codecov plugin? IMO this is a trivial case..

only if you feel like it.

d-v-b · 2026-07-23T19:26:28Z

+]
+
+
+class TestArraySpecHashEq:


i don't know why we need a class here. IMO plain functions and data are simpler

d-v-b · 2026-07-23T19:26:44Z

+]
+
+
+# Mutations: each mutate kwargs to an uneqal version


Suggested change

# Mutations: each mutate kwargs to an uneqal version

# Mutations: each mutate kwargs to an unequal version

d-v-b · 2026-07-23T19:27:59Z

+        base = _make_spec(**kwargs)
+        variant = _make_spec(**{**kwargs, **mutate(kwargs)})
+        assert base != variant
+        assert hash(base) != hash(variant)


hashes collisions between unequal objects can occur, right? so maybe we just need to check that they are unequal, and not that the hashes differ?

d-v-b · 2026-07-23T19:29:11Z

can you update the stale comment in sharding.py or re-enable the LRU cache?

fix(ArraySpec): proper and robust equality semantics for ArraySpec by…

ff62fb2

… checking fill_value for byte-identicality, fixes zarr-developers#3054.

github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jul 23, 2026

fix: added extra case for unequal types

55f096e

d-v-b reviewed Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(ArraySpec): proper and robust equality semantics for ArraySpec by checking fill_value for byte-identicality, fixes #3054.#4183

fix(ArraySpec): proper and robust equality semantics for ArraySpec by checking fill_value for byte-identicality, fixes #3054.#4183
sehoffmann wants to merge 2 commits into
zarr-developers:mainfrom
sehoffmann:fix/3054

sehoffmann commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

sehoffmann commented Jul 23, 2026

Uh oh!

d-v-b commented Jul 23, 2026

Uh oh!

d-v-b Jul 23, 2026

Uh oh!

d-v-b Jul 23, 2026

Uh oh!

d-v-b Jul 23, 2026

Uh oh!

d-v-b commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	# Mutations: each mutate kwargs to an uneqal version
	# Mutations: each mutate kwargs to an unequal version

Uh oh!

Uh oh!

Conversation

sehoffmann commented Jul 23, 2026

Summary

Behavioral Changes (summarized):

For reviewers

Author attestation

TODO

Uh oh!

codecov Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sehoffmann commented Jul 23, 2026

Uh oh!

d-v-b commented Jul 23, 2026

Uh oh!

d-v-b Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

d-v-b Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

d-v-b Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

d-v-b commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jul 23, 2026 •

edited

Loading