Hesela / Research library
Research & analyses
Storage engineering, examined through sources, measurements and reproducible methods.
133 analyses · Page 4 of 5
Newest firstNAND paired pages: why a later write can damage earlier data
Two NAND pages can share cells. Distinguish interrupted programming, neighbor interference and the protection promised by an SSD.
NAND read-retry: when a successful SSD read takes longer
Why retry counts do not determine SSD read latency: chip evidence, pipelined recovery and reproducible synthetic stage-timing examples.
Perseus: why slow drives are not always broken drives
Perseus detects drive slowdowns, but shared-path faults can escape its peer model. Review host scheduling, benchmark limits and checked ratio examples.
RAID rebuild risk: read-error models, field evidence and resilver boundaries
A datasheet error-rate limit is not a measured rebuild-failure probability. Under an illustrative independent-error model, reading five 12 TB survivors at 1 error per 10^14 bits gives 99.2% for at least one read error; five 20 TB survivors at 1 per 10^15 bits give 55.1%. Those are different read volumes, 60 TB and 100 TB, and neither result estimates the chance of losing an entire array. Rebuild, resilver, spare activation and checksum verification are separate recovery milestones.
RAID write hole: why a bitmap is not a write journal
A bitmap tracks dirty regions; PPL repairs a narrower parity hazard; a journal preserves replay data. A reproducible XOR example separates these guarantees.
RAID write penalty is not a speed ratio
Read-modify-write, reconstruct-write and full-stripe accounting: why four or six member operations do not mean one-quarter or one-sixth of full-stripe speed.
SMART replacement prediction: recall, F-score and random splits
Botezatu et al. report 98% Seagate recall; their Hitachi table gives 79%, while the prose says 81%. These random-split results are not a prospective deployment estimate.
SMART zeros, missing attributes and failure-day selection
A complete Q4 2025 raw-data audit separates 66 failed drives with five measured zeros from 170 with partial zero-only readings, and explains why complete-case filtering changes the manufacturer mix.
SMR stalls, cache limits and the errors SMART can miss
An SMR performance stall and a failed write are different observations. Research explains how internal cleaning can delay work; historical WD40EFAX reports document command errors despite a passing SMART health result. Neither establishes that cache exhaustion caused every reported error, nor that SMR has a higher fleet failure rate.
Storage fencing: why a lost heartbeat does not stop writes
Delayed I/O, SCSI and NVMe reservations, host identity and fencing tokens explain why path failover alone cannot protect shared storage.
Storage timeouts: an expired timer is not a cancelled write
Distinguish drive recovery limits, CDL descriptors, host timeouts and multipath queueing; preserve retry identity when write completion is uncertain.
Tape shoe-shining: how much a backhitch costs, what speed matching removes, and what the logs do not show
Why tape backhitches interrupt streaming, when filemarks confirm buffered writes, and how host supply and explicit byte units change a buffer-drain estimate.
USB storage resets: recovery is not a root-cause diagnosis
A UAS abort or USB reset identifies a recovery action, not the failed component. Pinned Linux code, a historical incident and SMART documentation separate transport symptoms, compatibility quirks and disk-health evidence.
What do public storage-incident postmortems show about drives that fail together?
Four public postmortems show drives failing together or in bulk because they shared something (a controller, a lot, a firmware), and the reports separate what the logs showed from what the operator only suspects. InfiniCLOUD (October 2026): 6 of 11 disks in a ZFS RAIDZ3 pool sat on one controller; one disk failed at 01:43 JST on 1 October, and a fourth had failed by 19:21 JST on 2 October, 41 h 38 min later, beyond the 3-disk tolerance; the cause of the last three failures is still unknown. Echelon VPS (February 2026): both NVMe drives of one mirror, same manufacturing lot and nearly equal write volume, hit a firmware assertion 4 min apart and took 218 instances offline for 94 min. GitHub (January 2016): a 2 h 6 min outage after a power blip, when over 25% of servers could not see their own drives after rebooting, a known firmware issue. Seagate (January 2009): a firmware event-log bug hung drives at power-up when the log counter was at entry 320 or 320 + n x 256, and the first fix was withdrawn after it disabled some 500 GB drives. None of the four reports publishes the raw logs, and three of the four name no drive or firmware version.
What smartctl and smartd can and cannot tell you about a failing drive
A successful smartctl process is not a complete health measurement. ATA exit bits distinguish command problems, threshold warnings and logged errors; PASSED is a reported check result, not a guarantee against failure. This guide separates those signals, explains smartd alert prerequisites and preserves unknown readings in tested collector examples.
Why a valid checksum can still return the wrong data
Checksums, identity, freshness, Linux verity and disk encryption: what successful verification does and does not prove, with reproducible examples.
ZFS and Btrfs checksum counters: coverage and blind spots
Checksum counters expose corrupt reads that device health can miss, but zero errors do not prove integrity or identify which component failed.
ZNS resource limits: why closing a zone may not unblock writes
Open and active zone budgets, Linux write plugging, native versus emulated Zone Append, and the limits of ZNS performance comparisons.
34.4% of RASR failures are not the SSD: 20.5% human mistakes, 13.9% cables, 31.2% labeled Failed Device
In Xu et al., USENIX ATC 2019, 34.4% of Reported As “SSD-Related” (RASR) failures are not caused by the SSD device. That 34.4% is human mistakes (20.5%) plus faulty cables (13.9%), inferred from which repair worked. It is not the complement of a measured device failure: Table 11 labels 31.2% Failed Device because replacing the SSD is the last resort, 11.9% Transient, and 22.5% Undetermined (FSCK 16.5% plus Data Check 6.0%). The paper prints 20.5 and 34.4. It does not print 22.5 or the addition 20.5 + 13.9.
38% of failed datacenter SSDs showed none of the four SMART symptoms; the ones that did failed 3 to 20 times as often
In Narayanan et al. (SYSTOR 2016), a study of over half a million SSDs in Microsoft datacenters over nearly 3 years, 62% of devices that fail-stopped had shown at least one of four SMART symptoms (uncorrectable or CRC data errors, sector reallocations, program or erase failures, and SATA downshifts) and 38% had shown none. Devices with a symptom failed 3 to 20 times as often as devices without one, with data errors the strongest at up to 20 times. Of the models whose failure rate could be compared with vendor figures, two consumer models exceeded the published 0.61% to 0.73% annual failure rate. A multi-factor classifier reached 87% precision and 71% recall, but it was scored with 5-fold cross-validation on one operator's data. Maneas et al. (FAST 2020) and Backblaze (2016) show the same pattern in other fleets: symptoms raise risk and a large share of failures give no warning.
AFR is a rate, not a one-year failure probability
Why exposure, zero-failure probability bounds, survival curves and censoring assumptions matter when interpreting drive reliability, with reproducible synthetic examples.
Azure disk-error prediction caught 36.5% of faulty disks at a 0.1% false-positive rate, and random cross-validation said 91.6%
In Xu et al. (USENIX ATC 2018), Microsoft's disk-error predictor, CDEF, found 36.50% of faulty disks on one test set when it was allowed to wrongly flag only 0.1% of healthy disks (29.67% to 41.09% across three test sets), with about 10,000 healthy disks per 3 faulty ones. Scored with a random split, the same task gave 91.64%, which is why the authors insist on testing forward in time. Adding system-level signals such as OS events to SMART raised the average catch rate at that false-positive rate from 27.6% to 30.3%, and to 35.8% with feature selection. In production at Azure the authors report about 63,000 fewer minutes of VM downtime a month. The data are one cloud system in one month, and labels come from engineers' root-cause analysis of service problems.
Disk failure prediction: signals, scores and alert costs
Lu et al.'s fleet study tested SMART, performance and location signals. Interpret its scores alongside explicit alert denominators, prevalence and validation limits.
Enterprise SSD replacements ran at 0.22% a year across 1.4 million NetApp drives, with firmware the biggest swing
In Maneas et al. (FAST 2020), about 1.4 million SSDs in NetApp enterprise storage systems had an average annual replacement rate (ARR) of 0.22%, with models ranging from 0.07% to nearly 1.2%. That is well below the 4% to 10% of Google's flash drives removed within four years and the 2% to 9% a year the same paper quotes for hard disks, though the definitions of replacement differ. Within the NetApp data, the earliest firmware versions of some models were replaced up to 8 to more than 10 times as often as later ones, and the chance of a second replacement in the same RAID group within a week was 9.39% against 0.0504% in a random week. One third of replacements were predictive and another third were severe SCSI errors. Method limits are on the note.
Fail-slow NVMe SSDs: what SMART and heartbeats miss
Separate fleet counts from model averages, review uncertain follow-up dates, and test how a live heartbeat can miss stalled storage work.
Google measured a disk MTTF of 10 to 50 years but a node MTTF of 4.3 months; 37% of node failures came in bursts
In Ford et al. (OSDI 2010), one year of data from tens of Google storage cells of 1000 to 7000 nodes each, the mean time to failure was 10 to 50 years for a disk, 10.2 years for a rack, and 4.3 months for a node, so node failures, not disk failures, drove availability. 37% of failures were part of a burst of at least 2 nodes. In the authors' model, a 10% cut in the disk failure rate raised stripe availability by less than 1.5%, while a 10% cut in the node failure rate raised data availability by 18%. Maneas et al. (FAST 2020, NetApp SSDs) and Schroeder and Gibson (FAST 2007, HPC1 disks) also found strong clustering of drive replacements in time, so a failure rate alone does not describe risk.
Media error or link error? What SCSI sense codes and Linux ATA log lines can and cannot tell apart
ATA registers, Linux's error mask and SCSI sense data describe different layers. They can distinguish a reported media error from a transfer error, but do not uniquely identify the failed component. Recovery thresholds produce conditional verdicts, not guaranteed actions: the audited libata path does not force a SATA ATA disk into PIO. A reproducible comparison also shows why decoding must be tied to a kernel revision.
Network is the largest fail-slow hardware type: 29 of 112 occurrences in 101 reports
Network is the largest hardware type in Table 3 of Gunawi et al., FAST 2018: 29 of 112 root-cause occurrences, ahead of CPU 25, disk 23, SSD 22, and memory 13. Those 112 occurrences come from 101 fail-slow reports at 12 institutions, incidents reported from 2000 through 2017, and a report can name more than one cause. Device errors account for 40 occurrences and firmware issues for 20. Temperature, power, environment, and configuration sum to 44 of 112 (39.3%), which rounds to the proceedings’ statement that 39% of root causes are external factors, not 39% of the 101 reports.
NVMe health log: warning bits, wear estimates and counter limits
Read NVMe health counters with the right units and version: wear is not failure probability, host writes are not NAND writes, and warnings are not a fault history.
Replacing disks at 200 reallocated sectors removed 88% of triple-disk RAID failures at one vendor
In Ma et al. (FAST 2015), about 1 million SATA disks in backup systems from one storage vendor showed that the count of reallocated sectors (RS) correlates strongly with whole-disk failure. Replacing a disk once its count passed 200 eliminated about 88% of the RAID failures caused by three-disk errors, which were 80% of all RAID failures, or about 70% of all disk-related incidents. The same paper says the signal is incomplete: in simulation, a threshold below 200 caught 52% to 70% of impending whole-disk failures, with 0.8% to 4.5% false positives. Google's FAST 2007 data and Backblaze's 2016 SMART tally point the same way on reallocated sectors, and both also show failed drives with no warning.
Looking for a technology? Browse technology cards or open the knowledge corpus.