Analyses / Data movement
Rsync after a one-byte shift: less payload, not free work
/ Reproducible software experiment
One inserted byte did not make rsync resend our 256 KiB basis file. Forced delta transfer reused all 262,144 original bytes and sent one literal byte. But the operation still exchanged signatures and references: one literal byte is not one byte of total traffic.
On this page6 sections
Fixed blocks do not require fixed search offsets
The original rsync algorithm partitions the receiver's existing file into blocks, then searches the sender's new file at arbitrary byte offsets. A cheap rolling checksum selects candidates; a second checksum confirms them. Matching blocks become references to the receiver's data; unmatched ranges become literals. [Algorithm]
In the twelve-byte example ABCD | EFGH | IJKL, prepending x moves the old blocks to offsets 1, 5 and 9. Splitting the new file only at offsets 0, 4 and 8 misses them. Sliding the search does not. This is different from choosing content-defined chunk boundaries.
The rolling recurrence removes the outgoing byte and incorporates the incoming one, avoiding a full window rescan at each offset. The report uses modulus 65,536; its checksum was inspired by Adler-32, not specified as identical to it. Its MD4 second checksum is historical, not a claim about every modern rsync session. [Recurrence]
What the native program actually reused
We ran six constructed inputs at block sizes of 512 and 4,096 bytes, with local defaults, explicit whole-file transfer and explicit delta transfer: 36 runs. Every destination exactly equalled its source, and every source remained unchanged.
| Edit | Block | Literal | Matched |
|---|---|---|---|
| Unchanged | 512 | 0 | 262,144 |
| Unchanged | 4,096 | 0 | 262,144 |
| Insert 1 byte at start | 512 | 1 | 262,144 |
| Insert 1 byte at start | 4,096 | 1 | 262,144 |
| Insert 1 byte inside | 512 | 513 | 261,632 |
| Insert 1 byte inside | 4,096 | 4,097 | 258,048 |
| Delete 1 byte inside | 512 | 511 | 261,632 |
| Delete 1 byte inside | 4,096 | 4,095 | 258,048 |
| Replace 1 byte inside | 512 | 512 | 261,632 |
| Replace 1 byte inside | 4,096 | 4,096 | 258,048 |
| Replace entire content | 512 | 262,144 | 0 |
| Replace entire content | 4,096 | 262,144 | 0 |
The prefix insertion preserved every old block. In contrast, an insertion 17 bytes inside an existing block broke that block: 513 literal bytes with 512-byte blocks, or 4,097 with 4,096-byte blocks. The literal count is therefore not simply the number of edited bytes.
Our independent, offset-aligned control reused zero complete blocks after the prefix insertion at either block size. That control compares exact bytes, without hashes. Rsync's full-file result also passed an exact byte comparison, so the distinction does not depend on assuming checksum equality means byte equality.
With wholly different content, delta mode reused nothing. This is a useful negative control: the method does not manufacture reuse where the chosen inputs provide none.
A smaller block can send fewer literals and more total bytes
For the interior insertion, the 512-byte configuration reported 2,640 sent plus 3,107 received bytes; the 4,096-byte configuration reported 4,432 plus 419. Thus the smaller blocks needed only 513 literals but a larger two-direction reported byte sum: 5,747 versus 4,851.
These are the program's local protocol counters, not packet capture or Ethernet accounting. They demonstrate why literal payload alone is an incomplete optimization target. They do not measure disk reads, physical writes, CPU time or wall-clock speed.
The manual distinguishes unmatched literal data from data reconstructed locally. Local-path transfers default to whole-file mode unless batch writing changes that default. In our selected-file controls, both local-default and explicit whole-file modes sent every source byte as literal data. [Manual]
Use a real transfer when measuring these counters: a dry run does not execute the data path. Separately account for file selection. We forced transfers with --ignore-times; the unchanged-file row is not a prediction of what an ordinary no-change backup would send.
The research frontier: reusing work already done below
SkySync, presented at FAST 2026, obtains suitable checksums from the storage layer and adapts them for delta generation. Its FSC prototype builds on librsync; the authors reimplemented the unavailable dsync prototype for their CDC baseline. [Paper, sections 3.3-4.1]
Evaluation combines constructed edits with five versioned datasets; reported points average ten runs. Figures 15-16 report client/server time reductions of 19.2-43.7%, or 16.2-36.4% with hardware assistance. These are the authors' testbed results, not a forecast for an installed rsync binary. Metadata extraction is measured separately in Figure 18.
The useful engineering question is whether compatible checksums can be obtained cheaply for the exact data ranges and versions being synchronized. Integration and extraction costs still matter. We inspected the published artifact, but did not run SkySync or reproduce its performance results.
Reproduce and interpret the observations
The basis is the first 262,144 bytes of SHAKE-256 applied to the ASCII string hesela-rsync-delta-v1. Interior edits occur at offset 131,089. The full replacement uses a different documented input string. Inputs are quiescent; each run starts with a fresh destination containing the basis.
The executable was rsync 3.2.7, protocol 31, on Linux tmpfs. We explicitly chose xxh128 and left transfer verification enabled. This non-cryptographic checksum selection is an experimental control, not authentication or deployment advice. No compression flag was used.
python3 -B observe.py --scratch /dev/shm --output replay.json
python3 -B -m unittest -v test_observe.pyDownload all 36 observations and raw outputs, the runner, control tests and reproduction notes. The JSON identifies the script and binary hashes, environment, commands and input recipe. Exact protocol overhead can vary between versions and invocation details; the byte oracle must still pass.
Synthetic software observations, not network throughput, media durability, production savings or a SkySync replication. All fixtures were forced into transfer. The installed version is identified, not recommended. There is one observation per deterministic configuration, not a sampled workload or a timing distribution requiring a confidence interval. Successful OS-visible reads do not establish power-loss durability.
Continue with delta synchronization, rolling hashes and why synchronization is not a restore test.
Sources
- Tridgell and Mackerras. The rsync algorithm. ANU TR-CS-96-05, June 1996 (HTML algorithm section).
- Tridgell and Mackerras: rolling checksum recurrence and modulus.
- Rsync 3.2.7 manual: whole-file, block-size, ignore-times, checksum-choice and statistics.
- Zhang, Li, Tang, Xue, Shu and Zhang. SkySync: Accelerating File Synchronization with Collaborative Delta Generation. FAST 2026, pp. 461-476, sections 3-4.
- SkySync authors' public implementation (not executed for this study).