HDF5 chunk cache: more memory does not fix every reread

The same 128 bytes, two different read counts

The fixture contains a 64 by 64 array of unsigned 32-bit integers, split into four 32 by 32 chunks. Each chunk expands to 4,096 bytes; gzip level 6 stores each in 1,325 to 1,335 bytes in this build. We request eight different scalars from each chunk: 32 selections and 128 application bytes in total.

Round-robin order visits A, B, C, D eight times. Grouped order visits all eight selected values in A before B, C and D. The selected coordinates and values are identical as a set; only their order changes.

With 17 slots and 16 KiB, the first visit to each chunk reads its stored payload; subsequent visits need no additional raw chunk read. With one slot, four times as many cache bytes still gives 32 reads. With grouped selections, one slot and only 4 KiB achieves the same four reads as the larger cache.

The compression ratio is not the cache-capacity ratio. Our 2 KiB case can hold any compressed chunk's byte length, but cannot retain its 4 KiB decoded image. It reads a chunk on every selection. The 4 KiB round-robin case fits one decoded chunk but cannot retain the four-chunk reuse set.

Bytes, slots and reuse distance are separate constraints

An HDF5 chunked layout partitions an array into blocks; filters operate per chunk. A small selection from a filtered chunk can require processing that whole chunk. This is different from content-defined deduplication chunks. [4]

The raw-data chunk cache has a byte budget and a hash table. Colliding entries cannot coexist even when unused byte capacity remains. The HDF Group also documents changes to chunk-index calculation across versions; a layout-specific modulo calculation should not be treated as a permanent API contract. [2]

We deliberately use one slot to force the collision boundary without assuming a particular hash formula. Seventeen slots is a control that worked for these four chunks, not a recommended universal value. The experiment keeps the eviction parameter rdcc_w0 at zero throughout, so it does not compare eviction policies.

Cache parameters are per dataset. The h5py guide describes different default byte budgets before and after HDF5 2.0; record effective values instead of assuming that a historical default applies. In this experiment, each open dataset's access property list must match the requested settings or the runner stops. [3]

What the instrument sees, and what it does not

The entire HDF5 file lives in a Python memory buffer. A wrapper logs each read and readinto callback during every scalar selection. Chunk offsets and stored sizes come from get_chunk_info_by_coord, not from assumptions about adjacent chunks. The runner counts only exact matches and rejects ambiguous partial overlaps. Metadata reads remain in the trace but are excluded from raw-payload totals. [5]

Each case gets a fresh HDF5 open and an empty dataset cache. All returned integers are checked against the original array. We repeat six cases three times on one host: 18 controlled case runs, not 18 independent performance samples. Raw observations retain all 576 selections, driver reads, cache settings, environment and source hashes.

No physical disk is read by the observation path. The 42,624 versus 5,328 byte totals refer to stored chunk payload requested through the file driver. They exclude metadata, Python overhead and copying, and cannot be called SSD traffic, filesystem IOPS or elapsed-time improvement.

Multiplying chunk reads by 4,096 gives an inferred decoded volume of 131,072 versus 16,384 bytes. Dividing by 128 selected bytes gives 1,024 versus 128. These are arithmetic decode-volume ratios for this filtered scalar trace, not measured decompressor calls or a universal read-amplification factor. Even the cache-friendly case decodes substantially more data than the application selects.

A historical case study, with a different measurement boundary

The HDF Group's 2015 technical paper studies a satellite-data array with gzip-compressed chunks larger than its configured cache. Pages 16-19 compare cache enlargement, chunk-aligned access and rewriting with smaller chunks. The reported timings belong to that application and environment. We did not reproduce its data or benchmark; our smaller in-memory experiment isolates callback counts and deliberately separates collision pressure from byte capacity. [1]

The paper also describes per-dataset memory accumulation. Enlarging every cache is therefore a resource decision, not a free optimization. Its historical slot-default and indexing discussion should not substitute for the installed library's settings. [1]

A practical order for investigating rereads

1. Record the array type, chunk dimensions, filters and exact selection trace. For fixed-width values, multiply dimensions by element size to obtain decoded chunk bytes.
2. Find the set of chunks revisited before eviction. In our round-robin trace that set contains four chunks, not one.
3. Check both byte capacity and slots. The one-slot control shows why increasing only bytes can have no effect.
4. Try grouping or batching accesses when application ordering permits. Our grouped test preserves the selected values but would need reordering if a consumer requires the original sequence.
5. Then measure the real application's latency, CPU cost and memory footprint on its real driver. Our callback counts alone cannot select a production cache size.

Changing the physical chunk layout requires a separate rewrite decision and validation of the resulting file. Faster reads do not establish application completeness or crash durability; those are covered in the HDF5 SWMR experiment.

Reproduce and inspect the trace

Download the runner, tests and pinned requirements together. The dataset page provides the CSV, complete JSON and field definitions. Original code and observations are CC0-1.0; upstream materials keep their licenses.

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r hdf5-chunk-cache-requirements.txt
.venv/bin/python test_hdf5_chunk_cache.py
.venv/bin/python hdf5-chunk-cache.py --repetitions 3 > observations.json

Measured environment: Python 3.12.3, h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3, zlib 1.3, Linux x86_64. File-format bounds are explicitly v110/v110. The harness uses no source data files or production datasets. Different library builds may produce different stored sizes or access behavior; investigate a failed control instead of overwriting the published expectations.

Primary sources

1. The HDF Group (2015): Improving I/O Performance When Working with HDF5 Compressed Datasets, pp. 7-20
2. The HDF Group: Chunking in HDF5, chunk caching and filters
3. h5py 3.15: File objects, Python file-like objects and chunk cache
4. h5py 3.15: Datasets, chunked storage and filter pipeline
5. h5py low-level API: DatasetID.get_chunk_info_by_coord

Sources checked 17 October 2026. This is a bounded engineering observation, not a novel cache algorithm.