Zarr archive checks: when missing chunks become plausible numbers

The same loss, two different denominators

The consumer experiment starts with eight dimensionless integers, 10..17, split into two four-element chunks with storage fill zero. It deletes the chunk containing 14..17, then opens a new Xarray reader. Dask and consolidated metadata are disabled; the rows below have no CF _FillValue attribute.

For format 2, decoded NaNs are skipped by the default floating-point mean: 46 / 4 = 11.5. For format 3, substituted zeros remain ordinary values: 46 / 8 = 5.75. The complete sum is 108. Neither a finite mean nor a count of eight establishes that eight measurements survived. [7] [8]

The written-zero control matters too: the intact array 0..7 loses its genuine zero from the default format-2 denominator. Value-based masking cannot recover why a zero appeared. Setting skipna=False exposes NaNs in masked cases but leaves the unmasked 5.75 result unchanged.

Labels and missing-data support answer different questions

Hoyer and Hamman's original Xarray paper distinguishes named dimensions and coordinates from arbitrary attributes, and describes missing-data handling in arithmetic, alignment and aggregation. [13] That design helps keep labels aligned; it does not establish that every expected stored measurement reached the reader. Our inference is to validate coordinate identity, storage completeness and decoded validity separately. A correctly labeled array can still contain substituted fill values. The 2017 software paper provides architectural context, not evidence for the later Zarr behaviors measured here.

Requested options need a consumer-path control

Both pinned Xarray releases ignored an explicit use_zarr_fill_value_as_mask override through the public open_zarr route. Direct backend controls did change masking. This reproduces known upstream issue #10269, not a new Hesela discovery. No first corrected release or current-Xarray behavior is asserted. [10]

Upstream merged the forwarding correction in PR #11748 on 17 October 2026. Our pinned releases precede that correction; the observations are not a claim about all later versions. Fixing option forwarding does not recover deleted values or distinguish intentional sparsity from loss: it changes how the reader interprets the same fill-valued input.

CF decoding is separate: with format-3 _FillValue=0, requesting no storage-fill masking still yielded missing values, including through the diagnostic backend route. Disabling mask_and_scale returned raw zeros but also disables other decoding behavior. The backend route is an experimental control, not a recommended stable API workaround.

The final store can lose the history

Zarr represents chunked arrays using metadata and stored objects. Its fill value supplies uninitialized regions; it is not automatically an error marker or validity mask. Intentional omission avoids storing uniform regions. [1] [2]

In Zarr 3.1.3, never writing the second chunk and writing then deleting it left identical remaining keys and file bytes, in both format controls. Both returned the same fill-valued tail. No inspection of those final stores alone distinguishes the tested histories. Completeness requires an expectation established outside that ambiguous state.

CRC32C rejected a one-byte mutation when the affected chunk was selected. A prefix-only read succeeded. Deleting the corrupted object made the full default read succeed with fill instead. Swapping whole valid chunks preserved their checksums while exchanging coordinates. The checksum did not malfunction: its protected payload lacked the application-level location binding. [4] This is a concrete example of the checksum and identity distinction.

Strict reads do not establish inner-shard completeness

Our separate Zarr-Python 3.4.1 format-3 experiment verifies effective read_missing_chunks=False policy. Shards bundle smaller read units inside one stored object; their index may validly mark an inner chunk empty, which decodes as fill. [3] [5]

Strict reads rejected a selected missing standalone object and a missing whole shard. They accepted a present shard with an empty inner tail. Stored all-fill data and an omitted inner chunk also returned identical values despite different inventories. Requiring every possible chunk would reject intentional sparsity unless the producer contract forbids it.

A valid older shard still passes

We saved a shard after writing only its first half, completed the second half, then restored the earlier object. Metadata stayed unchanged; index and stored-payload checksums stayed valid. Strict reads returned fill in the tail. The final store matched the intentionally sparse control. An object count cannot distinguish them; a trusted key-to-content inventory from the completed generation detects the changed shard.

This is deliberate replacement with a saved object, not a demonstrated cloud rollback, checksum collision or security flaw. The inference is that format validity does not establish freshness or intended completion.

Establish the effective configuration

For our existing local array, passing configuration to open_array(..., mode="r") did not apply it in 3.4.1: the existing-array branch does not forward those creation arguments. The documented with_config route did, as did tested zarr.open and scoped-global routes. Eight controls record requested and effective flags separately. [6]

array = zarr.open_array(path, mode="r").with_config(
    {"read_missing_chunks": False}
)
if array.config.read_missing_chunks is not False:
    raise RuntimeError("Required read policy was not applied")

This configures a handle, not an archive validator. A default reopen resumed fill behavior: this runtime policy was not persisted in metadata. Exercise a known missing-object control through the actual consumer, then read the required scope.

Old metadata can expose new values

Consolidated metadata reduces lookups, but does not freeze chunk values. In Zarr 3.1.3 format 2 we consolidated an eight-element array, retained a reader, grew the array to twelve, changed its first value to 999 and added calibration. [11]

A new unconsolidated reader saw current metadata. Re-consolidation refreshed a new reader but not the retained handle. Neither operation establishes an atomic capture across concurrent writers. These measurements are format-2 only, not format-3 consolidation or cloud-consistency evidence.

An archive acceptance contract

These recommendations are engineering inferences from the controls, not a new Zarr requirement or a general-purpose validator.

1. Specify intended content. Record array paths, shapes, types, coordinates, units and intentional sparse or invalid regions.
2. Capture one coherent generation. Coordinate writers or use an explicit publication protocol. A hash list collected across unrelated updates is not a consistent snapshot.
3. Bind location to bytes. Retain trusted expected keys and content digests, including metadata and deliberate omissions. Hashing after an unknown loss does not recover that expectation.
4. Exercise the actual consumer. Record effective policy, format, versions, CF attributes and decoding settings. Test complete, sparse, missing and substituted inputs.
5. Validate interpreted results. Check coordinate relationships and value validity alongside count, sum and mean. Count still includes substituted zeros.

Related mechanisms appear in HDF5 external dependencies and explain why backup verification is not a restore test. No decoder option here reconstructs deleted measurements.

Evidence, versions and reproduction

The companion dataset provides generated CSV, metadata and an exact-byte reproduction package. Three studies remain separate: 54 original Zarr observations; 252 Xarray observations per release, 504 across two releases; 138 strict-policy reads plus eight routing and one reopen controls. These are different observation units, not a pooled failure sample.

Each package retains original harnesses, tests, raw JSON, dependency pins and provenance. Publication tables are derived from JSON, with source-hash verification and recomputed arithmetic. Use separate Python environments for the pinned stacks. Original observations remain unchanged from 19 and 23 October 2026.

A 25 October replay reproduced all 558 older-study observations in both normal and optimized Python, with every recorded field equal except the explicitly documented Python patch version (3.12.14 to 3.12.3). Installed dependency versions and relevant source-module hashes matched. The separate 3.4.1 suite also reproduced its shard, routing and reopen controls. These checks strengthen repeatability without adding independent failure samples.

Scope: tiny synthetic int32 arrays in local stores and one Python implementation family. No field loss frequency, cloud race, crash, power failure, Dask or current-Xarray behavior was measured. Xarray is a consumer over the same Zarr implementation, not an independent implementation. Repetition checks reproducibility, not population uncertainty. This is source/code self-review, not independent peer review.

Original code and observations are CC0-1.0; third-party libraries and sources retain their licenses and are not redistributed.

Primary sources

1. Zarr format 2 specification: chunks and fill values
2. Zarr format 3 core specification: fill values and array reads
3. Zarr sharding codec 1.0: empty inner chunks and index layout
4. Zarr CRC32C codec 1.0
5. Zarr-Python: runtime array configuration
6. Zarr-Python 3.4.1: existing-array open and creation branches
7. Xarray 2025.10.1: open_zarr decoding contract
8. Xarray 2025.10.1: DataArray.mean and skipna
9. Xarray issue #10269: fill-mask option forwarding
10. Xarray v2026.09.0: Zarr backend implementation
11. Zarr-Python: consolidated metadata and synchronization
12. Xarray PR #11748: forwarding correction merged 17 October 2026
13. Hoyer and Hamman (2017), xarray: N-D labeled Arrays and Datasets in Python, JORS 5:10; pp. 3-4, DataArray and Missing Data