Research notes / Deduplication
Content-defined chunking: boundaries are not identity
12 October 2026 / Literature review and exact byte examples
Choosing where a chunk ends and deciding whether it already exists are separate operations. Content-defined boundaries can preserve reuse after an insertion; a matching boundary fingerprint does not establish matching content.
One inserted byte, three lost matches
Start with the twelve ASCII bytes ABCDEFGHIJKL and split every four bytes. Insert x at the beginning, then split again from offset zero. The original bytes are still present in order, but none of the new chunks equals an original chunk. This is an exact constructed example, not a storage trace.
| Input change | Four-byte grouping | Bytes in chunks equal to an original chunk |
|---|---|---|
| Original | ABCD | EFGH | IJKL | 12 |
| Insert x at the start | xABC | DEFG | HIJK | L | 0 |
| Insert WXYZ at the start | WXYZ | ABCD | EFGH | IJKL | 12 |
The last row is the counterexample to an overstatement: inserting a multiple of the block size preserves alignment. Repeated content can also yield matches after a one-byte shift. Therefore, fixed-size chunking can lose downstream matches; it does not always lose every match.
The byte count tests equality against the original chunk set. It is not saved capacity, a deduplication ratio or a timing result; metadata and index costs are absent. Illustration source.
A local anchor can survive a global offset change
LBFS selects boundaries using Rabin fingerprints of moving content windows, rather than fixed offsets. Unchanged regions can therefore recover the same boundaries after an edit. Its paper also describes the exception: minimum-size suppression and maximum-size forced cuts can disturb this synchronization. CDC is not a guarantee that an insertion changes exactly one chunk. [1, sections 3.1.1-3.1.2]
Keep three questions separate: which bytes form a chunk; whether those bytes match stored content; and where that content is physically placed. The last question explains why a good reuse rate need not mean a fast restore. See the companion restore-locality analysis.
FastCDC changes the boundary-selection cost
FastCDC combines Gear-based boundary tests, skipping before the minimum length and normalization. Normalization makes cuts harder before a target length and easier after it; it does not make all chunks equal-sized. [2, section 4]
The 2016 evaluation compares specified Rabin, Gear and AE implementations on seven datasets, using an i7-4770 and also an i7-930. Those are experimental conditions, not a portable speed multiplier. We do not reproduce its timings. [2, section 5]
An easily missed benchmark boundary: the file
Gregoriadis and colleagues' 2024 manuscript evaluates throughput, reuse and chunk-size distributions using four real-world collections and random data. Its throughput test reads from a RAMdisk and omits whole-chunk fingerprinting. Ten runs per configuration and a warm-up support its timing comparison, not a backup service throughput claim. [3, sections V-VI]
More subtly, the study concatenates each collection into one stream. Chunks can cross original file boundaries, unlike an implementation that resets at each file. The authors explicitly warn that even identical files may fail to match under that setup and exclude the terminal chunk from evaluation. A comparison that silently changes file framing can change its answer before the hash algorithm changes. [3, section V-B]
This makes an actionable review question: does the test process individual files, a concatenated stream, archive bytes, compressed bytes or encrypted bytes? Write that down alongside the dataset version. File names alone do not identify the input byte stream.
The rolling fingerprint is not the chunk ID
Restic documents Rabin-based chunk boundaries and separate SHA-256 references for the resulting plaintext blobs. Its repository format also has encrypted pack files and authentication. These are distinct layers; a boundary condition should not be reused as evidence that two entire chunks are identical. [4]
Likewise, our four-byte example uses direct equality, not a hash. It demonstrates an alignment effect without assuming collision resistance. Neither the example nor the reviewed speed tests establish protection against corruption or an adversary. Those require a separate threat model and implementation review.
Copied backups can carry incompatible chunk boundaries
Restic's repository-copy documentation gives a practical consequence: copying snapshots between repositories with different chunker parameters can undermine deduplication between their snapshots. For a newly initialized destination it documents --copy-chunker-params; existing repository parameters cannot be changed in place. Copying is also a read-and-write transfer under different encryption keys, not a free server-side move. [5]
Our inference is broader than that one tool: record the chunker version, parameters and file-framing rule when comparing two backup histories. A changed chunk layout can change the reusable units even when the underlying file bytes are identical. Do not infer corruption from lower reuse alone.
What a useful evaluation should report
Hesela's proposed protocol records input hashes, version history, insert/delete operations, min/target/max lengths, algorithm revision, compiler and CPU, and whether deduplication state is cold or already populated. Measure chunking CPU time separately from content hashing, index lookup, compression and storage I/O.
Report the observed size distribution and fraction of chunks forced by limits, not just a requested average. Report reused logical bytes and physical stored bytes with explicit denominators. Restore time remains a separate test. Include tiny files, repetitive data and end-of-file tails as correctness cases.
No CDC benchmark was run for this article. The executable examples establish only their stated byte-grouping results; the paper reviews explain mechanisms and methodological limits, not a current-product ranking.
Corpus: content-defined chunking, FastCDC, rolling hash and deduplication.
Primary sources
- Athicha Muthitacharoen, Benjie Chen and David Mazieres. A Low-bandwidth Network File System. SOSP 2001, sections 3.1.1-3.1.3.
- Wen Xia, Yukun Zhou, Hong Jiang, Dan Feng, Yu Hua, Yuchong Hu, Yucheng Zhang and Qing Liu. FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication. USENIX ATC 2016, pp. 101-114, sections 4-5.
- Marcel Gregoriadis, Leonhard Balduf, Bjorn Scheuermann and Johan Pouwelse. A Thorough Investigation of Content-Defined Chunking Algorithms for Data Deduplication. arXiv:2409.06066v1, 2024, sections V-IX.
- Restic project. Design: Backups and Deduplication; repository format. Documentation displayed as version 0.19.1.
- Restic project. Working with repositories: Ensuring deduplication for copied snapshots. Documentation displayed as version 0.19.1.
Reviewed 12 October 2026. The 2024 comparison is cited as version 1, not as an unverified later publication. Source works retain their own rights.