HDF5 SWMR: a readable file is not a complete measurement

What SWMR keeps coherent

HDF5 organizes typed arrays, attributes and groups; the application decides how they represent a scientific record. A dataset is not automatically a transaction. [6]

Single-writer/multiple-reader access orders metadata flushes so that a reader does not follow a pointer to a child structure that has not been flushed. It depends on the underlying system satisfying the required POSIX write semantics. This protects HDF5's internal structure, not an arbitrary relationship such as “every timestamp has a calibrated value.” [2]

The h5py workflow creates objects before switching the writer into SWMR mode. Readers open with swmr=True; the writer flushes datasets and readers refresh them. Notifications are an application responsibility. Neither a notification nor one successful refresh describes every dataset in the file. [1]

A metadata retry is not a drive-health event

On a non-atomic I/O path, a concurrent reader may encounter inconsistent checksummed metadata. HDF5 can retry that read within a configured attempt limit. H5Fget_metadata_read_retry_info reports those retries; H5Pset_metadata_read_attempts controls the limit. These are library metadata observations, not NAND read-retry counters or proof of a failing drive. Increasing the budget cannot validate a scientific record. Our experiment does not measure these counters. [7]

Plan the object schema before enabling SWMR

The documented SWMR scope permits updates to existing dataset payloads, but not adding or deleting objects. It also excludes modification or append operations on variable-length, string and region-reference datatypes. An acquisition design that creates a new group for every measurement therefore needs a different layout or access workflow; switching on SWMR alone does not make that design valid. These are documented restrictions, not extra cases tested by our fixed-integer experiment. [2]

Keep a file-format compatibility policy separate from the publication protocol. The h5py guide warns that the newer SWMR format cannot be opened by older, pre-SWMR libraries. Record the actual reader and writer library versions; the latest label is relative to the installed library, not a permanent format version. [1]

Two arrays, nine observable checkpoints

We used synthetic pairs (10, 100) and (20, 200), stored separately in x and y. A third dataset, published, holds a count chosen by the application. These integers are test inputs, not measurements from a sensor.

A pipe pauses the live writer at each checkpoint while a separate spawned reader refreshes and reads. The reader opens once and stays open. All 3 repetitions produced the same nine outcomes. The pauses make intermediate states reproducible; they do not measure how often a production reader would encounter them.

After the second x flush, the reader sees lengths 2 / 1 while both reads succeed. After the second y flush, lengths are 2 / 2, but the application has still published only one pair. Equal lengths therefore do not establish the chosen publication boundary either.

This is expected intermediate visibility, not evidence of an HDF5 defect. The engineering risk is treating a readable structure as a complete application record.

A publication watermark needs an immutability rule

Our demonstration appends and flushes both payload arrays before advancing the count. The reader refreshes the count first, captures n, refreshes both arrays, checks that each has at least n rows, and consumes only those rows. It rejects the deliberately invalid count of three when only two pairs exist.

The hidden requirement is that published rows remain unchanged. In the final checkpoint, we intentionally overwrite the first published x with 999, leaving its partner at 100. The count and length checks still pass. A watermark limits the selected prefix; it does not preserve an older version or detect a semantically mismatched pair.

This is a bounded application example, not a proven general-purpose commit protocol. It assumes one writer, monotonic publication during normal operation, immutable published rows, successful payload flushes before publication, and compatible file/driver semantics. A reader must handle temporarily unavailable payload and validate its record schema. Our writer is paused during every observation, so unpaused races and torn marker updates are outside the experiment.

For instrument acquisition, define which arrays, units and calibration version form one record before deciding what “available” means. For mutable records, a count alone cannot supply a consistent historical view. Compare database snapshots and application-consistent backups.

A scientific schema supplies relationships, not a commit

The H5MD paper provides a concrete example: its time-dependent elements associate step, time and value datasets, with matching leading dimensions. It also permits shared time coordinates through hard links. These conventions describe what related samples mean. [8]

Our inference is that a live acquisition application needs both such a schema and an explicit publication rule. A hard link shares an object; it does not atomically advance that object together with every value array. We did not test an H5MD implementation, and the paper is not evidence for SWMR transaction guarantees.

An unlocked file is not permission for a second writer

HDF5's documented scheme separates OS-level file locking from persistent superblock flags. Entering SWMR writing removes the file lock while leaving the relevant writing flags. The absence of a held lock does not turn SWMR into independent multi-writer access. Disabling locking does not provide a replacement coordination protocol. Parallel HDF5 is a separate, coordinated multi-process model. [4]

After an unclean exit, a leftover superblock mark may prevent an ordinary open. The official h5clear documentation explicitly limits its role: clearing such a mark is not general corruption repair. An operator should preserve evidence and establish that no writer is active before considering recovery actions. We did not run that tool; the separate process-exit experiment below leaves the file unchanged. [5]

Flush visibility does not prove power-loss recovery

H5Dflush flushes dataset buffers without evicting their contents. H5Drefresh refreshes the dataset's buffers. The API documentation notes that HDF5 does not control every buffering layer. These operations do not, by themselves, establish an all-or-nothing application transaction across multiple datasets. [3]

Both experiments can read from operating-system cache. We did not trace persistence syscalls, interrupt power, or examine recovery after a kernel crash or reboot. Nothing here establishes that a scalar watermark update is failure-atomic. A durability claim requires its own driver, filesystem and device evidence, as in the crash-test oracle discussion.

After the writer exits, its status flags can remain

A second, separate experiment compares normal close with os._exit(73), which ends the writer without HDF5 close or Python cleanup. Each case creates a new private file, flushes both payload arrays and optionally publishes one pair. The parent waits for the writer to exit before attempting fresh ordinary and SWMR read-only opens. This tests a selected process-exit boundary, not an interruption in the middle of an I/O request.

The two unclean SWMR exits retained 0x05: write-access bit 0 and SWMR-write bit 2. The non-SWMR control retained only 0x01 and rejected both opens. Normal close left 0x00 and both opens succeeded in this environment. The format specification defines these bits as cleared at final close. They are file metadata, not a liveness probe or a measurement of corruption. [9]

In the unpublished SWMR case, both payloads were readable but the selected prefix remained empty. In the published case it contained [[10,100]]. A readable payload therefore still does not establish that the application published it. Conversely, an ordinary-open error did not prove that the writer was still alive: this harness had already joined the process.

Boundary: all exits occurred after explicit HDF5 flush calls. No power loss, kernel failure, reboot, torn write, concurrent writer or repair operation was tested; OS and device caches remained available. These 12 case runs are controlled demonstrations, not a failure-rate sample. Do not interpret their successful reads as a power-loss guarantee or use them to justify clearing flags on a production file.

The separate raw observations include full payloads, errors, environment and source/requirements hashes. Run the process-exit harness with the same pinned requirements as the first experiment:

.venv/bin/python hdf5-process-exit.py --repetitions 3 > process-exit.json

Reproduce the observation

The dataset page provides CSV, raw JSON, environment and source hash. Download the Python harness and pinned requirements into one directory, then run:

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r hdf5-swmr-requirements.txt
.venv/bin/python hdf5-swmr.py --repetitions 3 > observations.json

The runner uses private temporary files only, bounded process waits and explicit runtime checks that remain active with Python optimization enabled. Linux, Python and GNU stat are required. Use --temp-dir to select a disposable test directory on the filesystem under examination.

Published environment: h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3, Python 3.12.3, Linux 6.8.0-146-generic, driver sec2. GNU stat reported ext2/ext3, a filesystem-magic label that does not distinguish ext-family versions. All datasets use one-element chunks without compression. libver="latest" is relative to the installed library; it is not a fixed interoperability target.

This is one host and 27 ordered checkpoints, not 27 independent reliability trials. No confidence interval, latency estimate, hardware-failure claim or general filesystem comparison is warranted.

Primary sources

1. h5py 3.15: Single Writer Multiple Reader
2. The HDF Group: SWMR metadata flush dependencies
3. The HDF Group: H5Dflush and H5Drefresh
4. The HDF Group: File locking and superblock flags
5. The HDF Group: h5clear is not a general repair tool
6. The HDF Group: HDF5 Data Model and File Structure
7. The HDF Group: H5Fget_metadata_read_retry_info and metadata read attempts
8. de Buyl, Colberg and Hoefling: H5MD, arXiv:1308.6382v2 (2014), sections 2 and 3.4.1
9. The HDF Group: File format, version-3 superblock consistency flags

Checked 17 October 2026. Documentation and measured library versions are recorded separately.