docs: test more code snippets in docs

.github/workflows/docs.yml

@@ -24,7 +24,10 @@ jobs:
           persist-credentials: false
       - uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
       - run: uv sync --group docs
-      - run: uv run mkdocs build
+      # --strict turns warnings into errors, so a docs code block that fails to execute
+      # at build time (e.g. a non-exec python fence disrupting a later exec="true" block)
+      # fails CI instead of merging as a silent warning.
+      - run: uv run mkdocs build --strict
         env:
           DISABLE_MKDOCS_2_WARNING: "true"
           NO_MKDOCS_2_WARNING: "true"

changes/4016.bugfix.md

@@ -0,0 +1 @@
+Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested.

docs/contributing.md

@@ -12,7 +12,7 @@ If you find a bug, please raise a [GitHub issue](https://github.com/zarr-develop
 
 1. A minimal, self-contained snippet of Python code reproducing the problem. You can format the code nicely using markdown, e.g.:
 
-```python
+```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable"
 import zarr
 g = zarr.group()
 # etc.
@@ -225,10 +225,10 @@ hatch --env docs run serve
 
 #### Adding executable code blocks in the documentation
 
-Zarr uses [Markdown Exec](https://pawamoy.github.io/markdown-exec/usage/) to execute code blocks in Markdown files. Add `exec="on"` to a code block header for it to be executed when the docs are built. For example:
+Zarr uses [Markdown Exec](https://pawamoy.github.io/markdown-exec/usage/) to execute code blocks in Markdown files. Add `exec="true"` to a code block header for it to be executed when the docs are built. For example:
 
 ````md
-```python exec="on"
+```python exec="true"
 print("Hello world")
 ```
 ````
@@ -253,6 +253,64 @@ renders as:
 print("Hello world")
 ```
 
+#### Validating code blocks: `exec` vs `test`
+
+Every Python code block in the documentation is checked by a test
+(`tests/test_docs.py`) so that examples cannot quietly rot — the bug that motivated
+this was an example calling `zarr.create_array(..., mode="w")`, an argument that does
+not exist, which went unnoticed because nothing ran it. A block declares *how* it is
+validated using one of two independent attributes:
+
+  - **`exec="true"`** — Markdown Exec runs the block **at docs-build time to render its
+    output** into the page. This is the attribute described above; it is also what the
+    test suite executes. Use it for ordinary examples whose output should appear in the
+    docs.
+  - **`test="true"`** — the block is **run by the test suite only**, *not* at build time.
+    Use this for an example that should be validated but cannot run in the docs-build
+    environment — for example one that needs a GPU or a cloud backend. Markdown Exec
+    leaves a `test="true"` block as a static, syntax-highlighted snippet (it never
+    executes it), while the test suite still runs it (see the marker note below).
+
+A block may carry both (`exec="true" test="true"`), though in practice `exec="true"`
+already implies it is tested, so you rarely need `test="true"` alongside it.
+
+The two attributes are kept separate on purpose: `exec=` controls *build-time rendering*
+and `test=` controls *test-time validation*. Tagging a GPU/cloud example `exec="true"`
+would make `mkdocs build` try to run it on a machine without that infrastructure and fail
+the build; `test="true"` lets it be validated without being built.
+
+##### Opting a block out of validation
+
+A handful of blocks genuinely cannot run and are not executable Python — a REPL
+transcript, a deliberately-incorrect "before" snippet, a `--8<--` file include. Mark
+these explicitly by opening the fence with
+`exec="false" reason="REPL output transcript, not executable source"` (supply a reason
+that fits the block).
+
+`exec="false"` with a non-empty `reason` is an explicit, greppable opt-out. A test
+(`test_no_unvalidated_blocks`) requires **every** Python block to be either `exec="true"`,
+`test="true"`, or `exec="false"` with a reason — so a block can never silently skip
+validation. A bare ` ```python ` fence, or a typo like `exec="on"`, fails that test.
+
+##### Marker-bound blocks (GPU, S3)
+
+A `test="true"` block that needs special infrastructure declares a pytest marker with
+`markers="..."`, which binds it to that infrastructure in the test suite:
+
+  - `markers="gpu"` — run only under `pytest -m gpu` (the GPU CI environment); skipped
+    elsewhere via `importorskip("cupy")`.
+  - `markers="s3"` — run against a mock S3 (moto) backend supplied by a test fixture, so
+    the example can use a bare `s3://…` URL with no test-only connection details on show.
+
+##### Placement of `test="true"` blocks
+
+Because Markdown Exec does not execute a `test="true"` (or `exec="false"`) block, placing
+one *before* an `exec="true"` block on the same page can disrupt the build-time execution
+of that later block. Put `test="true"` blocks **after** all `exec="true"` blocks on the
+page (or on a page where they are the only Python block). The `test_test_only_blocks_come_last`
+test enforces this, and the CI docs build runs with `--strict` so any such breakage fails
+the build rather than passing as a warning.
+
 #### Building documentation without executing code blocks
 
 Sometimes, you may want the documentation to build quicker. You can disable code block execution by commenting out the [markdown-exec plugin](https://github.com/zarr-developers/zarr-python/blob/884a8c91afcc3efe28b3da952be3b85125c453cb/mkdocs.yml#L132) in the mkdocs configuration file. This will make code blocks and cross references render incorrectly (i.e., expect build warnings), but also reduces build time by ~3x. Be sure to undo the commenting out before opening your pull request.

docs/quick-start.md

@@ -127,18 +127,6 @@ be done in a separate step.
 Zarr supports persistent storage to disk or cloud-compatible backends. While examples above
 utilized a [`zarr.storage.LocalStore`][], a number of other storage options are available.
 
-Zarr integrates seamlessly with cloud object storage such as Amazon S3 and Google Cloud Storage
-using external libraries like [s3fs](https://s3fs.readthedocs.io) or
-[gcsfs](https://gcsfs.readthedocs.io):
-
-```python
-
-import s3fs
-
-z = zarr.create_array("s3://example-bucket/foo", mode="w", shape=(100, 100), chunks=(10, 10), dtype="f4")
-z[:, :] = np.random.random((100, 100))
-```
-
 A single-file store can also be created using the [`zarr.storage.ZipStore`][]:
 
 ```python exec="true" session="quickstart" source="above"
@@ -173,4 +161,18 @@ z = zarr.open_array(store, mode='r')
 print(z[:])
 ```
 
+Zarr also integrates seamlessly with cloud object storage such as Amazon S3 and Google
+Cloud Storage using external libraries like [s3fs](https://s3fs.readthedocs.io) or
+[gcsfs](https://gcsfs.readthedocs.io):
+
+```python test="true" session="s3demo" markers="s3" source="above"
+import zarr
+import numpy as np
+
+z = zarr.create_array(
+    "s3://example-bucket/foo", shape=(100, 100), chunks=(10, 10), dtype="f4"
+)
+z[:, :] = np.random.random((100, 100))
+```
+
 Read more about Zarr's storage options in the [User Guide](user-guide/index.md).

docs/user-guide/arrays.md

@@ -619,7 +619,7 @@ Without the `shards` argument, there would be 10,000 chunks stored as individual
     Because the feature is still stabilizing, it is disabled by default and
     must be explicitly enabled:
 
-    ```python
+    ```python exec="true" session="arrays-rectilinear"
     import zarr
     zarr.config.set({"array.rectilinear_chunks": True})
     ```

docs/user-guide/cli.md

@@ -45,9 +45,13 @@ This will write new `zarr.json` files to `input.zarr`, leaving the existing v2 m
 
 To open the array/group using the new metadata use:
 
-```python
+```python exec="true" session="cli-open" source="above"
 import zarr
-zarr_with_v3_metadata = zarr.open('path/to/input.zarr', zarr_format=3)
+
+# create a small array to open (stands in for the migrated store)
+zarr.create_array("data/cli-demo.zarr", shape=(4, 4), chunks=(2, 2), dtype="i4", overwrite=True)
+
+zarr_with_v3_metadata = zarr.open("data/cli-demo.zarr", zarr_format=3)
 ```
 
 Once you are happy with the conversion, you can run the following to remove the old v2 metadata:

docs/user-guide/data_types.md

@@ -360,7 +360,7 @@ print(type(a.dtype))
 
 But if we inspect the metadata for the array, we can see the Zarr data type object:
 
-```python
+```python exec="false" reason="REPL output transcript, not executable source"
 type(a.metadata.data_type)
 <class 'zarr.core.dtype.npy.int.Int64'>
 ```

docs/user-guide/examples/custom_dtype.md

@@ -2,6 +2,6 @@
 
 ## Source Code
 
-```python
+```python exec="false" reason="pymdownx snippet include directive, not python source"
 --8<-- "examples/custom_dtype/custom_dtype.py"
 ```

docs/user-guide/gpu.md

@@ -16,7 +16,7 @@ Zarr can use GPUs to accelerate your workload by running `zarr.Config.enable_gpu
 [`zarr.config`][] configures Zarr to use GPU memory for the data
 buffers used internally by Zarr via `enable_gpu()`.
 
-```python
+```python test="true" session="gpu-demo" markers="gpu" source="above"
 import zarr
 import cupy as cp
 zarr.config.enable_gpu()

docs/user-guide/performance.md

@@ -204,7 +204,7 @@ determines the maximum number of concurrent I/O operations.
 The default value is 10, which is a conservative value. You may get improved performance by tuning
 the concurrency limit. You can adjust this value based on your specific needs:
 
-```python
+```python exec="true" session="perf-concurrency"
 import zarr
 
 # Set concurrency for the current session
@@ -234,7 +234,7 @@ By default it is `None`, which lets Python choose the pool size (typically
 
 You can set it explicitly when you want more predictable resource usage:
 
-```python
+```python exec="true" session="perf-workers"
 import zarr
 
 zarr.config.set({'threading.max_workers': 8})
@@ -260,7 +260,7 @@ For example, if you're running Dask with 10 threads and Zarr's default concurren
 
 **Recommendation**: When using Dask with many threads, configure Zarr's concurrency settings:
 
-```python
+```python exec="false" reason="requires dask, which is not in the docs test environment"
 import zarr
 import dask.array as da
 

docs/user-guide/v3_migration.md

@@ -39,7 +39,7 @@ the following actions in order:
    - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs
      directly from `numcodecs`:
 
-     ```python
+     ```python exec="false" reason="intentionally shows the old/incorrect import for contrast"
      from numcodecs import Blosc
      # instead of:
      # from zarr import Blosc

pyproject.toml

@@ -204,6 +204,10 @@ list-env = "pip list"
 template = "test"
 extra-dependencies = [
     "universal_pathlib",
+    # Needed so tests/test_docs.py is collectable under `pytest -m gpu`; otherwise its
+    # module-level importorskip("pytest_examples") skips the whole module and the gpu
+    # docs example is never executed on GPU hardware.
+    "pytest-examples",
 ]
 features = ["gpu"]
 
@@ -277,9 +281,8 @@ readthedocs = "rm -rf $READTHEDOCS_OUTPUT/html && cp -r site $READTHEDOCS_OUTPUT
 [tool.hatch.envs.doctest]
 description = "Test environment for validating executable code blocks in documentation"
 features = ['remote']
-dependency-groups = ['test']
+dependency-groups = ['remote-tests']
 extra-dependencies = [
-    "s3fs>=2023.10.0",
     "pytest-examples",
 ]
 
@@ -446,6 +449,7 @@ filterwarnings = [
 markers = [
     "asyncio: mark test as asyncio test",
     "gpu: mark a test as requiring CuPy and GPU",
+    "s3: mark a test as requiring a (mock) S3 backend via moto",
     "slow_hypothesis: slow hypothesis tests",
 ]
 

tests/conftest.py

@@ -531,3 +531,31 @@ def deep_nan_equal(a: object, b: object) -> bool:
     if isinstance(a, Sequence) and isinstance(b, Sequence):
         return all(deep_nan_equal(a[i], b[i]) for i in range(len(a)))
     return nan_equal(a, b)
+
+
+# Shared mock-S3 (moto) backend. A single server is reused across the whole test session by
+# every test that needs S3 -- both the fsspec store tests and the documentation examples --
+# instead of each module standing up its own. Consumers create their own buckets and choose
+# how the endpoint reaches the client (explicit storage_options vs. the AWS_ENDPOINT_URL
+# env var) on top of this fixture.
+MOTO_SERVER_PORT = 5555
+MOTO_ENDPOINT_URL = f"http://127.0.0.1:{MOTO_SERVER_PORT}/"
+
+
+@pytest.fixture(scope="session")
+def moto_server() -> Generator[str, None, None]:
+    """Start a session-scoped moto S3 server and yield its endpoint URL.
+
+    importorskip lives inside the fixture so moto is only required when a test actually
+    requests an S3 backend, not for the whole test session."""
+    moto_server_mod = pytest.importorskip("moto.moto_server.threaded_moto_server")
+
+    server = moto_server_mod.ThreadedMotoServer(ip_address="127.0.0.1", port=MOTO_SERVER_PORT)
+    server.start()
+    # moto needs *some* credentials present; use throwaway values if the environment has none.
+    os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "foo")
+    os.environ.setdefault("AWS_ACCESS_KEY_ID", "foo")
+    try:
+        yield MOTO_ENDPOINT_URL
+    finally:
+        server.stop()