Slow RAID hours are not request tail latency
Published 2026-10-12 · updated 2026-10-15
A slow-drive hour, a slow-RAID hour and a request percentile count different events. This source review separates them and shows why fan-out estimates need explicit dependence and workload assumptions.
- Observed unit
- Hourly average latency of data drives, relative to the RAID-group median; not a per-request percentile. [Hao, section 3]
- Observation window
- 87 days on average across RAID groups, not a common 87-day panel; Table 1 spans 1-1470 days for HDD groups and 1-94 for SSD groups.
- Workload boundary
- Mostly full-stripe I/O, proprietary filesystem/controller, predominantly non-rotating-parity RAID-6; parity drives excluded. [Hao, sections 3.1-3.3]
- Model boundary
- Independent equal-probability leaves give 1-(1-p)^n. Reusing this formula requires a common event, time window and wait-for-all completion rule.
Source figures, separate populations
Historical source extract: hourly observations, request experiments, report counts and detected-case arithmetic. Each row retains its own population and unit.
| Measure | Value | Unit | Population or scope | Source |
|---|---|---|---|---|
| Data disks studied | 458,482 | drives | NetApp data drives; group observation lengths vary | Source 1 |
| Data SSDs studied | 4,069 | drives | NetApp SSD data drives; parity drives excluded | Source 1 |
| Disk drive hours observed | 857,183,442 | drive hours | Same study | Source 1 |
| SSD drive hours observed | 7,481,055 | drive hours | Same study | Source 1 |
| Disk hours at least 2x slower than the RAID group median | 0.22 | percent of drive hours | Disks, slowdown ratio of 2 or more | Source 1 |
| SSD hours at least 2x slower than the RAID group median | 0.58 | percent of drive hours | SSDs, same rule | Source 1 |
| Slow hours with a 1.5x threshold | 0.69 to 1.27 | percent of drive hours | Disks (0.69) and SSDs (1.27) | Source 1 |
| RAID hours with at least one slow disk | 1.54 | percent of RAID hours | Disk RAID groups, 2x threshold | Source 1 |
| RAID hours with at least one slow SSD | 2.23 | percent of RAID hours | SSD RAID groups, 2x threshold | Source 1 |
| RAID hours with a slow drive, 1.5x threshold | 4.6 to 4.8 | percent of RAID hours | Disks (4.6) and SSDs (4.8) | Source 1 |
| Slow disks per RAID hour with a slow disk | 1.70 | slow drives per tail hour | Derived: 1,885,804 / 1,109,514; conditional mean, not a dependence test | Source 1 |
| Slow SSDs per RAID hour with a slow SSD | 1.80 | slow drives per tail hour | Derived: 43,016 / 23,964; conditional mean, not a dependence test | Source 1 |
| Slow drives that stay slow for more than an hour | 35 to 40 | percent of slow drives | SSDs (35) and disks (40) | Source 1 |
| Drives with at least one slowdown | 26 to 29 | percent of drives | Disks (26) and SSDs (29) | Source 1 |
| Disk tail hours if the longest tail were cut | 0.6 | percent of RAID hours | Opportunity estimate; assumes a faster parity read, not measured request p99 | Source 1 |
| Requests slow when each server is slow 1 time in 100, fan-out 100 | 63 | percent of requests | Idealized service, Dean and Barroso | Source 2 |
| Requests slow when each server is slow 1 time in 10,000, 2,000 servers | 18 | percent of requests | Derived rounded arithmetic, not a measured service rate | Source 2 |
| 99th percentile latency of one random leaf request | 10 | milliseconds | A large fan-out Google service, measured at the root | Source 2 |
| 99th percentile latency until all leaf requests finish | 140 | milliseconds | Same service | Source 2 |
| SSD read latency increase from garbage collection | 100 | times | Solid-state storage with modest write activity | Source 2 |
| Cut in 99.9th percentile read latency from tied requests | 38 | percent | BigTable, mostly idle cluster; tied replica request after 1 ms with cancellation | Source 2 |
| Fail-slow hardware incident reports studied | 101 | reports | Gunawi et al., collected from large-scale clusters | Source 3 |
| Reports with months-long detection time | 17 | percent of reports | All 101 reports; detection time unknown in 45% | Source 3 |
| Detected fail-slow cases relative to monitored drives | 0.12 | percent of drives | Derived: 304 / 248,000 over 10 months; not incidence per drive-hour | Source 4 |
The 24-row legacy extract is retained. Paired HDD/SSD values in range columns are category summaries, not confidence intervals. The companion JSON supplies integer denominators and deterministic calculations; decimals describe arithmetic precision, not measurement certainty.
Method
Source audit of four full papers, with locators in the table notes and accompanying dataset. Published rounded figures are retained separately from Hesela's reproducible arithmetic. Table 1 of Hao et al. supplies the integer exposure and slow-hour counts; no per-request traces were obtained. Synthetic examples use explicit values, exact fractions and nearest-rank percentiles. They are counterexamples, not a replication of a production fleet.
Limits
Hourly summaries cannot recover request percentiles, simultaneous sub-hour stalls, or an empirical confidence interval that accounts for temporal and group dependence. A conditional mean above one establishes multiple slow classifications within some group-hours, not statistical dependence between drives. Historical fleets, operator reports and selected benchmark traces must not be pooled. No current SSD failure rate, universal RAID slowdown, measured mitigation benefit or vendor ranking follows.
What the evidence supports
Read the denominator first. Hao et al. classify hourly averages, not individual slow requests. Their 0.22% HDD drive-hour fraction and 1.54% HDD RAID-hour fraction describe different sampling units. For the mostly full-stripe workload studied, a straggler can delay completion; not every RAID read waits for every member.
Multiplicity is not a dependence test. Dividing slow drive-hours by tail group-hours gives the conditional mean count of slow drives. The companion calculations reproduce 1.70 for HDD and 1.80 for SSD. Even independent Bernoulli events can produce a conditional mean above one; a joint, workload-matched model is needed before calling that excess correlation.
Opportunity is not a measured guarantee. The paper's 0.6% HDD residual tail-hour estimate describes removing the longest data-drive tail, assuming reconstruction can use a faster parity drive. It is not a demonstrated reduction of request p99 on arbitrary RAID implementations. Its ToleRAID experiments separately use injected delays. [Hao, sections 4.1.2 and 5.2]
Other evidence answers other questions. Dean and Barroso's request-level fan-out examples and BigTable tied-request results are not NetApp measurements. Tied requests include cross-server cancellation; duplicating reads without that mechanism can have different costs. [Dean and Barroso, Tables 1-2]
Gunawi et al. collected 101 reports, not 101 uniformly sampled devices or distinct incidents: repeated same-root-cause occurrences within an institution can share one report. Detection time was unknown in 45% of reports. These records support mechanisms, not a population incidence rate. [Gunawi, sections 2 and 3.5]
Perseus reports 304 detected cases during ten months monitoring 248,000 drives; 0.12% is only a crude detected-case proportion. Its separate benchmark includes 315 verified positives among about 41,000 drives over 15 days, with normal peers selected from the same clusters. Do not substitute that enriched benchmark for fleet prevalence, or either denominator for drive-hours. [Lu, abstract and section 5.1]
Background on drive types is on the technology index . Warning signs before a failure are also covered in the SMART failure signals analysis.
What an hourly mean cannot reveal
Synthetic counterexample: 1,000 requests at 10 ms each versus 990 at 1 ms and 10 at 901 ms. Both means are 10 ms. Percentiles use the nearest-rank rule: sorted observation ceil(q * N). Neither row is a device benchmark.
| Scenario | Mean | p99 | p99.9 |
|---|---|---|---|
| Uniform | 10 ms | 10 ms | 10 ms |
| Bursty | 10 ms | 1 ms | 901 ms |
A second blind spot is the reference itself. Three peers at 10, 10 and 10 ms all have ratio 1 to their median. At 100, 100 and 100 ms, all ratios remain 1. The relative detector is unchanged although every latency increased tenfold. This arithmetic example is not a claim about how often common-mode stalls occur.
Fan-out depends on the joint events
If X counts slow required leaves, independence with equal marginal probability p gives P(X > 0) = 1 - (1-p)^n. Without independence, equal marginals alone give p ≤ P(X > 0) ≤ min(1, np). These are mathematical bounds, not confidence intervals.
| Per-leaf p | Leaves | Independent | Possible bounds |
|---|---|---|---|
| 1.00% | 100 | 63.40% | 1.00% to 100.00% |
| 0.01% | 2000 | 18.13% | 0.01% to 20.00% |
For n = 100 and p = 0.01, E[X | X > 0] = np / P(X > 0) is 1.577, already greater than one under independence. Thus the paper-count ratios 1.70 and 1.80 alone cannot diagnose correlated drives. Here 1% (all slow events coincide) and 100% (mutually exclusive slow events) are both possible union probabilities at the same 1% marginal. Changing dependence while holding marginals fixed need not increase the union probability.
Reproduce and apply
Integer counts reproduce the HDD percentages. For SSDs, 100 * 43,016 / 7,481,055 is 0.5749991144%, slightly below the 0.575% rounding boundary; the source prints 0.58%. The legacy extract retains 0.58% as reported, while generated results retain the count-based quotient. We do not infer a different underlying population from this discrepancy.
For an operational investigation, retain request-level histograms or traces alongside per-drive absolute latency, workload mix, timestamps and group membership. Record which responses actually gate completion and the cost of redundant reads. Compare RAID tail hour,tail latency and Perseus slowdown ratio before combining their metrics.
Sources
- 01Hao et al., The Tail at Store, USENIX FAST 2016, pp. 263-276; Table 1, sections 3, 4.1.2, 5.2 and 6 · accessed 2026-10-15
- 02Dean and Barroso, The Tail at Scale, CACM 56(2), 2013, pp. 74-80; Tables 1-2; DOI 10.1145/2408776.2408794 · accessed 2026-10-15
- 03Gunawi et al., Fail-Slow at Scale, USENIX FAST 2018; report collection (section 2) and detection time (section 3.5) · accessed 2026-10-15
- 04Lu et al., Perseus, USENIX FAST 2023, pp. 49-64; abstract, sections 4.4-4.5 and 5.1 · accessed 2026-10-15