HDF5 virtual datasets: an intact file can return missing or different data

The file stores a mapping, not necessarily the values

An HDF5 virtual dataset (VDS) joins selections from other datasets into one array. A program can slice it like an ordinary dataset, even when its sources occupy separate files. Sources can be declared before they exist. This is useful for assembling independently produced arrays without first duplicating their contents. [1]

HDF5 explicitly permits missing sources to appear as fill values. The default fill value is zero, interpreted for the datatype; an application can choose another value. This behavior is a feature, not evidence of a library defect. The mistake is assigning a stronger meaning to a successful read than the mapping promises. [2] [3]

Why keep a virtual view instead of a copy?

ArrayBridge (Xing et al., ICDE 2018) used VDS to expose independently written files as one array and to reconstruct versions from shared data. Its coordinator combined mapping records before creating the view; repeatedly rebuilding accumulated mappings incurred quadratic mapping work. This is an interoperability and write-organization benefit, not a guarantee that one file contains an archive. [5]

The evaluation used SciDB 15.12 on NERSC Edison with Lustre. Its performance measurements belong to that configuration; our small fixture does not reproduce them. Our archival inference: removing data duplication makes preservation of the referenced data more important, not less.

A mapping includes source and destination selections, not just filenames. A hyperslab specifies logical array regions; it is not an on-disk chunk. Record the selections as well as the identities of source objects. Our raw fixture logs source names and paths, while its fixed row selections are specified in the downloadable harness.

Eight cases, with the virtual file unchanged

We mapped a.h5:/data and b.h5:/data into a fixed array of two rows and three columns. The first source held [11,12,13]; the second normally held [21,22,23]. These are dimensionless test integers, not instrument readings. Two VDS objects used the same mapping with different fill policies.

Removing the second file and removing only its expected /data dataset produced the same fill output. With default fill, that output was indistinguishable from the separate control containing genuine zeros. All arrays still reported shape (2,3). When only the VDS was relocated, both rows became fill values; relocating both sources with it preserved the original values.

The explicit sentinel made absence visible in these cases. It did not expose a substituted, structurally compatible source: both fill policies returned [91,92,93]. A recognizable missing-value code is useful, but it is not source authentication.

Even the neighboring source hashes can pass

The prefix-override case left view.h5, a.h5 and b.h5 untouched. We created a different b.h5 in another private directory and set HDF5_VDS_PREFIX for the new reader process. The reader obtained replacement values, even though a manifest checker examining only the original files beside the VDS would accept all three.

This matches the documented lookup order: for relative source names, environment prefixes are tried before a dataset-access prefix and the VDS file's directory; further fallback can use the relative name itself. Resolution behavior is part of the read environment, not merely the stored filename. [3]

Our inference: a reproducible archive needs both dependency identity and controlled resolution. The hashes did not fail to detect changed bytes. They correctly identified files that were no longer the complete set of files governing this read. This is a path-selection problem, not a cryptographic collision.

What a restore test should establish

Inspect Dataset.is_virtual and virtual_sources() to discover VDS mappings; the latter exposes source names and selections. Those records describe dependencies, not their current availability or authenticity. [4]

1. Inventory the intended data. Record source files, dataset paths, selections, shapes, types, units and the expected missing-data policy. Our two-file fixture is not a recursive dependency scanner.
2. Capture a stable set. Preserve the required source versions together. Hashes taken during unrelated concurrent updates do not establish a consistent scientific snapshot.
3. Control the reader environment. Restore in isolation from the originals and unintended search locations. Record or deliberately clear prefix settings and use a known working directory. Verify the identities of the sources actually selected.
4. Check values and validity. Do not accept a correct shape or an error-free read as completeness. Apply the scientific schema and missing-value rules to the required array regions.

A materialized ordinary array is one option when self-contained delivery matters more than avoiding duplication. Our control copied the full array while both sources were present and checked against the fixture, then stayed unchanged through all source manipulations. Materializing after a missing-source read could instead preserve the fill values as ordinary data. The control copies this array only; it is not a general HDF5 export preserving every attribute, group or relationship.

This extends the distinction between backup verification and an actual restore. It differs from SWMR publication consistency: our writers were closed, yet file dependencies still changed what the reader obtained. The analogous archival question for compression is whether the required dictionary is preserved and identified.

Method, scope and reproduction

Each of eight scenarios starts from fresh private files and is repeated three times: 24 case runs, each reading two VDS arrays and one ordinary control. All handles close before files are changed. Every final observation runs in a fresh process with an empty working directory. Relocation removes the original fixture directory; prefix overrides are absent except in the named case. Full arrays, mapping records, hashes and process outcomes are retained.

Measured environment: Python 3.12.3, h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3; Linux, sec2 driver and explicit v110 format bounds. This is a controlled API demonstration on one host, not a failure-rate estimate. We did not test power loss, concurrent mutation, damaged sources, permission failures, network filesystems, unlimited selections, nested VDS or external links. A different source failure may raise an error.

The dataset page provides CSV, JSON, column definitions and provenance. Download the harness and pinned requirements into one directory:

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r hdf5-vds-requirements.txt
.venv/bin/python hdf5-vds.py --repetitions 3 > observations.json

The harness uses only private temporary files and bounded subprocess waits. Its runtime checks also execute under optimized Python. Original code and observations are CC0-1.0; upstream documentation and libraries retain their own licenses.

Primary sources

1. h5py 3.15.0: Virtual Datasets
2. The HDF Group: Introduction to the Virtual Dataset
3. The HDF Group: H5Pset_virtual source-file resolution and H5Pset_fill_value
4. h5py 3.15.0: Dataset.is_virtual and Dataset.virtual_sources
5. Xing et al. (ICDE 2018): ArrayBridge: Interweaving declarative array processing in SciDB with imperative HDF5-based programs, sections V and VI

Documentation checked 17-18 October 2026. Measured library versions are stated separately.