LZ4 frame checks: block order, missing blocks and completion

Move the block and its checksum together

Our input is 65,536 ASCII A bytes followed by 65,536 ASCII B bytes. The compressor creates two independent blocks, each with a block checksum. We move or remove a whole block record, including its original checksum. No checksum is recalculated. The expected restored object always remains A followed by B, exactly 131,072 bytes.

Three independently generated frames isolate the controls: block checksums alone, block plus content checksums, and block checksums plus a declared uncompressed size. All three unchanged controls restore the intended object.

Order and inventory are separate checks. Block-only decoding accepts both B+A and A alone as complete frames. Declaring the original size rejects removal, but not same-length reordering. The content checksum rejects both changes in this fixture. Flipping one bit inside the first compressed payload triggers a block-checksum error, providing a corruption-detection control.

The format explains the difference: a block checksum covers that block's stored data; the optional content checksum covers the decoded sequence. Neither is a cryptographic authenticator. The independence flag does not authorize a caller to rearrange an object's contents. [1]

Returned bytes are not a completion signal

Removing only the last byte of the four-byte EndMark produces the entire expected A+B payload, yet the decoder still requests input. A consumer that checks only output length or equality and then silently accepts EOF misses this unfinished frame. Our acceptance rule rejects it.

The public API returns zero when a frame finishes; a non-error, positive return is a next-input hint. It also reports how many input bytes it consumed. Those are distinct signals. Appending another complete frame makes the first call sequence finish with input remaining. Our deliberately single-frame consumer rejects that extra input. A multi-frame application must instead continue according to its explicit framing and inventory policy. [2, LZ4F_decompress]

Do not publish partial output as a verified restore before the required checks finish. Our JSON counts output from successful calls only; bytes and sizes from an error-returning call are not treated as usable output. The chosen 8,192-byte output buffer affects when an error is encountered, not the expected object's identity.

Independence removes history, not every dependency

LZ4 independent blocks do not require preceding blocks' decoded history. They may still need an external dictionary: the frame specification initializes each independent block with the same dictionary when one is used. Its identifier is not the dictionary bytes. Our experiment uses no dictionary. [1, Dictionary ID]

A raw LZ4 block is a different interface from a frame. Its decoder needs bounds such as compressed size and a maximum decoded size supplied by the surrounding application. An arbitrary byte offset is not a block boundary. [3, Metadata]

Hesela's engineering conclusion: random access also needs an offset map, the correct decoder context and a verification policy for the requested data. A frame-wide content checksum cannot be fully checked by decoding only one selected block. Index design and per-object identity belong in the storage system around the codec; independent blocks alone do not supply them.

Compression ratio does not predict read cost

Ji and colleagues' FAST 2021 FPC study is useful context, not an LZ4 benchmark. Their prototype used LZO on F2FS, Android 9.0 and Linux 4.9, on a HiKey 960 with 4 GB RAM and 32 GB UFS. It reorganized read-critical blocks before compression. Foreground evaluation replayed captured writes; background evaluation profiled five cold launches per application before compression. [4, sections 2.2, 3.3 and 5.1]

The reported launch-time reduction averaged 5.2%, with two of ten applications slightly slower. The best case, 14.8%, was not the average. Its decompression amplification ratio counts compressed logical blocks co-located in physical blocks read, divided by logical blocks actually needed. It is not automatically the ratio of compressed bytes fetched to requested bytes. [4, section 5.2 and figure 13]

For a new design, measure useful bytes, bytes fetched, bytes decoded, cache state, CPU time and request latency separately. The checks above establish validation behavior, not a faster block size or a recommended codec.

Reproduce the twelve cases

Run the CC0 Python harness with Python 3 and an installed liblz4 shared library:

python3 lz4-frame-contract.py

The recorded JSON includes runtime versions, input and frame hashes, harness hash, exact-output comparisons, errors and generation time. The harness asserts its controls. There is no random seed because the payload and transformations are fixed. This is a bounded fixture parser, not a production decoder or a hostile-input test.

No checksum-collision probability, performance gain or fleet failure rate is inferred. The mutations demonstrate specified validation boundaries on one library build; they are not a newly discovered LZ4 vulnerability.

Continue with compressed-archive restore checks, LZ4 frames and block-independent compression.

Primary sources

1. LZ4 Frame Format 1.6.4, pinned to v1.10.0: block independence, checksums and dictionaries
2. LZ4 v1.9.4 public frame API: preferences and LZ4F_decompress completion contract
3. LZ4 v1.9.4 block format: out-of-band metadata and decoding bounds
4. Ji et al. Pattern-Guided File Compression with User-Experience Enhancement for Log-Structured File System on Mobile Devices. FAST 2021, pp. 127-140