Compressed archives: intact bytes can still be unreadable

A dictionary ID is not a dictionary

Zstandard permits independent frames and optional external dictionaries. The dictionary bytes are supplied separately; a frame's dictionary identifier does not contain them. That identifier can also be absent, so its absence does not prove the frame is self-contained. The optional content checksum stores the low 32 bits of XXH64 over decoded data. It is not an archive inventory or cryptographic authentication. [2]

The CLI exposes dictionary selection through -D, checksum selection through --check and --no-check, and identifier omission through --no-dictID. We deliberately exercise those switches below. Defaults and options should be recorded, not inferred from a filename. [3]

Fourteen cases: what did the check cover?

Our dictionary is 8,192 deterministic synthetic bytes. The 2,048-byte test payload is a slice of that dictionary, intentionally making dictionary dependence strong. The wrong dictionary inverts every bit without changing its length. These are raw-content dictionaries, not trained dictionaries with embedded identifiers. The harness checks that the produced frames omit a dictionary ID.

Wrong dictionary, successful process: with the checksum absent, the wrong dictionary produced 2,048 bytes and exit code zero, but those bytes differed from the expected payload. Enabling the checksum detected this particular mismatch. This is deliberate misuse, not a newly discovered codec defect or a measurement of checksum collision probability.

Complete frame, incomplete archive: we separately compressed record-one\n and record-two\n, then concatenated the two frames. Removing the second frame left a valid first frame. Decoding succeeded and yielded 11 bytes instead of the intended 22. There was no remaining malformed frame for the decoder to reject. The expected full-stream hash and length exposed the loss.

Complete output, failed process: removing only the final byte from the two-frame stream produced all 22 expected bytes on this build, but the decoder reported premature end and exited with code 1. A pipeline that accepts output while ignoring the producer's status can therefore accept a failed decode. The amount emitted before an error is implementation-dependent; the nonzero status must not be discarded.

The dictionary-free control occupied 2,061 compressed bytes; the dictionary-dependent checked frame occupied 21. That is deliberately contrived reuse, not a compression benchmark. It excludes the 8,192-byte dictionary and says nothing about net savings for a real collection.

Matching dictionary IDs can still return wrong bytes

The earlier cases omit IDs. This separate experiment keeps them: a 2,048-byte formatted dictionary is trained from 128 deterministic synthetic samples with ID 123456. The payload is its final 256 bytes. One variant changes only the ID to 123457; another flips one bit in the dictionary's final content byte while retaining the original ID and entropy tables. These are controlled mutations, not naturally occurring collisions.

Different label, unchanged coding data: changing only the dictionary ID caused rejection even though the frame had no content checksum. Omitting the frame ID then allowed that renamed dictionary to decode the same payload correctly. ID comparison and content-checksum comparison are separate checks.

Same label, changed coding data: the one-bit content mutation retained the expected ID, but returned a 256-byte payload differing from the expected bytes with exit zero when the checksum was absent or explicitly ignored. Verifying the checked frame rejected this mismatch. The successful controls used the original dictionary with and without a checksum. We did not find or search for a checksum collision.

The dictionary ID is therefore a selection guard, not a content hash or proof of authenticity. The CLI manual assigns responsibility for locally unique IDs to the dictionary manager; RFC 8878 supplies the format, not a dictionary retrieval protocol. [3] [2]

Rotation must preserve older dependencies

Our engineering inference: keep an immutable mapping from each archive generation to the exact dictionary digest. Do not overwrite a dictionary object in place or reuse its ID while retained archives still depend on it. Test recovery with an empty cache and the retained mapping. A trusted digest detects our changed dictionary; a matching numeric ID alone does not. The digest's trusted association matters as much as its algorithm.

This is a deliberately dictionary-dependent input, not evidence of failure frequency, compression quality or a codec defect. The mutation is confined to generated temporary files. Different readers and versions need their own conformance checks.

A stored checksum is not an executed check

RFC 8878 allows decoders to ignore checksums. Our CLI comparison makes that choice explicit: the same checked frame and wrong raw dictionary failed under --check, but returned wrong bytes and exit zero under --no-check. The frame still contained its checksum. Verification was disabled at read time, not omitted at write time. [2]

We also flipped one bit in the final checksum byte of the dictionary-free control, leaving compressed data unchanged. Verification rejected it; ignoring the checksum returned all 2,048 correct payload bytes. This isolates a checksum discrepancy from payload damage. It does not recommend suppressing verification, nor establish that arbitrary damaged files are recoverable.

Correct payload, changed metadata

A skippable frame carries user metadata that a normal Zstandard decoder passes over. We prefixed the checked control with generation=0001, then changed it to the equally long generation=0002. Both streams returned the same 2,048-byte payload with a successful checksum check and identical decoded SHA-256, despite different compressed-stream hashes. This is metadata exclusion, not a hash collision. [2]

The table's Payload accepted column intentionally retains the original narrow oracle. It cannot attest to skipped metadata. When such metadata matters, verify it separately or bind the complete stored object to a trusted digest. The existing compressed-object identity check below would detect our mutation; checking only the decoded length and hash would not.

An overlooked FAST lesson: preserve the decoding path

Bryan Ford's FAST 2005 VXA paper packaged decoders with compressed content and ran them in a restricted x86-32 environment. Its six-codec evaluation used kernel sources and media files on an Athlon 64 system. Figure 7 measured user-mode CPU time, not end-to-end restore time. The reported slowdown reached 11% relative to native x86-32 after a Vorbis adjustment; it is not a modern hardware estimate. [1]

Section 2.3 makes a practical distinction: testing a native decoder does not test an archived fallback decoder. Section 2.5 limits the proposal to decoding compressed streams, not preserving every document's semantics. We reviewed the paper, but did not run VXA or audit its historical sandbox. The lesson here is to test the actual future recovery path, not to execute old embedded code without isolation. [1]

Random access is another contract

The Zstandard seekable extension splits content into independently compressed frames and appends a seek table in a skippable frame. Entries give compressed and decompressed sizes, with optional checksums. This supports locating a range without decoding everything before it; it is additional structure, not a property guaranteed by any file ending in .zst. [5]

Our experiment uses ordinary concatenated frames, not that extension. We did not test seek-table validation, memory limits or encrypted archives. The formatted-dictionary experiment above does not exercise seek tables. A selected-range read would also need a separate coverage argument before being called a whole-archive test.

A stronger restore acceptance rule

The following checklist is Hesela's engineering inference from the experiment, not a universal archival standard. A trusted manifest should describe the intended object generation and complete content, not be reconstructed from the damaged input during the restore.

1. Retain the exact dictionary bytes and bind them to the archive by a cryptographic digest, alongside decoder version, options and required runtime. A short dictionary ID is a lookup hint, not a replacement for that evidence.
2. Verify expected compressed-object identity before decoding, then verify the full decoded length and digest against independently retained expectations. Protect the manifest itself if malicious replacement is in scope.
3. Record and test the decoder checksum policy, including a known-bad-checksum control. Require a successful decoder exit as well as matching output. Do not publish partial output from a failed restore as a valid recovered object.
4. Test in an isolated environment without relying on a production dictionary cache. Check application-level usability after decoding; matching bytes alone does not prove the recovered system works.

This extends the distinction in backup verification versus restore testing and the identity limits of checksums.

Reproduce without touching stored data

Download the Python harness and inspect the raw JSON observations. Python's standard library and an installed zstd CLI are sufficient:

python3 zstd-archive-contract.py --output observations.json

For the eight formatted-dictionary cases, use the separate identity harness and identity observations:

python3 zstd-dictionary-identity.py --output identity.json

That runner records training parameters, the length-delimited training-corpus hash, all dictionary variants, payload hashes and frame IDs. It uses one training thread and a 30-second limit per subprocess. Its measured environment is *** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***, Python 3.12.3, Linux x86_64. Repeated observations matched on this build; identical trained bytes across different versions are not promised.

The original runner creates private temporary dictionaries, sends bounded synthetic inputs through pipes and limits each subprocess to 15 seconds. It compares complete output bytes directly; published hashes make those comparisons independently inspectable. Runtime checks remain active under Python optimization. No existing data files are used as test inputs.

Measured environment: *** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***, Python 3.12.3, Linux x86_64. Documentation is pinned separately to Zstandard 1.5.7. The fourteen cases are deterministic demonstrations on one build, not fourteen independent reliability trials. They do not establish corruption frequencies, performance, power-loss behavior or general guarantees for other libraries.

Primary sources

1. Bryan Ford: VXA: A Virtual Architecture for Durable Compressed Archives, FAST 2005, sections 2 and 5
2. RFC 8878: Zstandard Compression, sections 3, 3.1.2 and 5
3. Zstandard v1.5.7 CLI manual: dictionaries, checksums and dictionary-ID omission
4. Zstandard v1.5.7 API manual: dictionary identification and decompression
5. Zstandard seekable format 0.1.0, pinned to v1.5.7

Sources checked 17 October 2026. All test data are synthetic; no external dataset is redistributed.