Storage latency: when the benchmark stops asking

One pause, two different questions

Consider one first-in, first-out server. Ordinary service takes 1 ms; the first request adds a 100 ms pause. Admit arrivals during [0, 2000) ms, then let every admitted request finish. There are no timeouts, drops, retries, caches or parallel workers.

The open stream sends at 0, 10, 20 ms and so on, independently of completions. The closed stream has one client: after each completion it waits 9 ms. Without the pause, both send every 10 ms. With it, the closed client's second request moves to 110 ms.

Hesela synthetic calculation, not SSD measurements. Each percentile uses all completed requests in that row's arrival cohort. Nearest rank selects sorted observation ceil(p x N); the two rows deliberately have different loads and denominators.

The open stream's first three response times are 101, 92 and 83 ms. Its second request arrives at 10 ms, starts at 101 ms, and completes at 102 ms. The closed stream records the first 101 ms response, but has no arrivals during that wait. Its other 189 responses take 1 ms.

This is a workload-model comparison, not an 83-fold device speedup. The closed result is useful for that one waiting client. It cannot answer what would happen if unrelated clients continued to arrive during the pause. That mismatch is where coordinated omission matters.

Write down where the clock starts

Keep four boundaries distinct: planned arrival, actual submission, service start and observed completion. In this example, arrival equals submission; service start can be later. A real generator can fall behind before submission too.

wrk2 measures from the planned send time to the response, explicitly retaining generator delay. Its maintainer also documents roughly millisecond timing granularity in that implementation. This is a useful measurement principle, not a recommendation to use an HTTP tool for microsecond SSD measurements. [3]

In fio, lat covers I/O creation to completion and combines slat and clat. Exact boundaries depend on the engine; async completion includes the point where fio reaps it. Do not rename any of these quantities as application end-to-end latency without checking the start event. [2]

What the paper supports

Schroeder, Wierman and Harchol-Balter compared open, closed and partly-open models using web/database testbeds and simulations. Their methodology holds service demands and load comparable while examining mean response time and scheduling. They show that arrival feedback can materially change conclusions, even at substantial concurrency. [1, sections 3-5]

That is not a universal theorem that open workloads always have worse tails: the paper also discusses scheduling-dependent exceptions. Its workloads are not modern NVMe characterization. Hesela's table is a separate deterministic illustration with unequal achieved load, not a re-creation of the paper's experiments.

A rate setting is not proof of independence

fio's offload mode separates submission work from completion processing, but adds overhead. The current option reference warns that it cannot reliably be used with asynchronous engines, while example jobs on the same page use libaio. Treat that documentation tension as a reason to validate the exact version and engine, not to copy an example as a guarantee. [2]

Also inspect achieved depth: the configured iodepth can differ from the observed distribution. A queue's capacity, an in-flight count and a count of waiting requests are different quantities. [2]

Hesela's practical check: impose a controlled pause on a disposable test target, retain scheduled/submitted/completed counts, and verify that the arrival policy and timestamps behave as claimed. Record generator CPU pressure and backlog. If it cannot sustain the offered stream, report that limit alongside the storage result.

Do not average the p99 column

A second exact counterexample uses two windows of 100 requests. Window A contains ninety-nine 1 ms responses and one 100 ms response; window B contains one hundred 100 ms responses. Their p99 values are 1 and 100 ms. Averaging gives 50.5 ms, but the combined 200-request p99 is 100 ms.

Combine observations or compatible histogram counts, then calculate the quantile. Retain units, precision, population and time-window boundaries. A percentile alone has discarded the distribution needed for pooling.

HdrHistogram can add modeled samples using an expected interval to compensate for missing observations. Those values are not newly measured I/Os. Label the correction and its assumptions; it cannot establish what unissued operations would actually have done on an overloaded device. [4]

A useful benchmark report

• State the arrival model, target rate, achieved submission/completion rates, concurrency and queue boundary.
• Define latency timestamps, units, quantile method and number of observations.
• Report errors, drops, timeouts and requests still outstanding at the cutoff separately from successful completions.
• Record workload mix, block size, read/write ratio, cache and persistence settings, device state, firmware and tool version.
• Preserve distributions and time series so a later reader can distinguish a short stall from a sustained regression.

Reproduce and inspect

No hardware was tested. All numbers above are deterministic outputs with fixed inputs and no random seed. They do not estimate a population or justify confidence intervals. The model excludes admission limits, real generator scheduling, device parallelism, caching and failure recovery.

Explore the dataset page with CSV, column definitions and hashes, or download the full synthetic request records and provenance, calculation module, tests and JSON generator. In their common directory, run node --test latency-measurement.test.mjs. Run the generator to refresh the records and source hashes; the timestamp records that actual run.

Corpus: coordinated omission, open-loop load, closed-loop load, queue depth and storage latency.

Related research: NAND read-retry latency and fail-slow drives and software scheduling. A measurement boundary does not identify the physical cause of a slow request.

Primary sources

1. Schroeder, Wierman and Harchol-Balter. Open Versus Closed: A Cautionary Tale. USENIX NSDI 2006, sections 2-5 and 7.
2. fio documentation: I/O depth, io_submit_mode and interpreting slat/clat/lat. Online manual reviewed 12 October 2026.
3. Gil Tene, wrk2: scheduled-arrival latency and coordinated omission; maintainer documentation, not an SSD benchmark.
4. HdrHistogram maintainers: corrected versus raw value recording and expected sampling intervals.

Sources reviewed 12 October 2026. No third-party traces or figures redistributed.