Follow Hesela
Updates
New and revised analyses, ordered by their recorded editorial date.
New publications RSS · Corrections RSS · All analysis dates RSS · Corpus updates
Site notes
: added the Hesela team information pages, contact directory, citation and correction controls, reading paths, corpus documentation and storage calculators.
Recorded changes
These notes preserve documented changes. Earlier history is incomplete; a revised date alone is not evidence of a correction.
- · correction
Datacenter failure tickets are not failed-device counts
Clarified counting units, operator-response endpoints and chained failure bursts after source review. Reported numeric values retained; the 2%/99% concentration remains source-reported, not independently reconstructed.
- · correction
Sector-error prediction: what adaptive scrubbing gains
Distinguished reported simulation results from deployment outcomes; added explicit budget arithmetic and corrected the percent-to-weeks interpretation. Original table values retained.
Latest publication dates
These are the latest dates recorded for each analysis, not a complete revision history. A build does not refresh them. Sources' access dates and dataset-generation timestamps are not used as editorial dates.
Updated
3D NAND retention: layer count is not a reliability rating
Separate NAND architecture, dwell, thermal annealing and read conditions before interpreting raw error rates or claims of restored SSD endurance.
Updated
AFR is a rate, not a one-year failure probability
Why exposure, zero-failure probability bounds, survival curves and censoring assumptions matter when interpreting drive reliability, with reproducible synthetic examples.
Updated
Atomic rename is not a durable commit
File contents, directory entries and commit acknowledgements have different persistence boundaries. Linux documentation and two USENIX studies explain what a rename does not guarantee.
Updated
Backup verification is not a restore test
Separate backup completion, integrity, freshness and usable recovery, with reproducible SQLite and rsync experiments and explicit limits.
Updated
Bloom filters: optimize the lookup, not each filter
LSM lookup costs, filter retention and counting limits, with reproducible allocation, deletion and memory examples grounded in primary sources.
Updated
eMMC, UFS and RPMB: interface, boot and security boundaries
How managed flash differs from raw NAND, why eMMC and UFS are not interchangeable interfaces, and why RPMB authentication is neither encryption nor a backup.
Updated
Ext4 journal replay: why old writes must sometimes be skipped
JBD2 replay, checksum failure paths and ambiguous log tails, with source-checked recovery boundaries and reproducible counterexamples.
Updated
Why retrying a failed fsync does not prove durability
Why a successful fsync retry can leave stale data: historical research, Linux close errors, POSIX portability, synchronization boundaries and reproducible descriptor-lifetime observations.
Updated
io_uring: completion is not a durable commit
io_uring write ordering, protection information, cancellation CQEs and linked timeouts: what each result proves, and what still needs synchronization.
Updated
iSCSI identity, sessions and recovery: what a second portal does not prove
Separate iSCSI names, portals, sessions, command ordering and recovery levels, with a source-checked distinction between projected and measured performance.
Updated
Magnetic retention: why 60 kBT is not an expiry date
A reproducible two-state model separates first reversal, final-state error and magnetization decay, and shows why a mean energy barrier cannot predict retention.
Updated
Disk failure prediction: signals, scores and alert costs
Lu et al.'s fleet study tested SMART, performance and location signals. Interpret its scores alongside explicit alert denominators, prevalence and validation limits.
Updated
Memory-mapped files: visibility, size and synchronization
Seven reproducible Linux checks distinguish mapped visibility, file size, partial-page tails, SIGBUS and sync argument errors from power-loss durability.
Updated
NFS stable writes: what COMMIT, flush and FUA actually promise
NFSv3 stability, COMMIT verifiers and Linux flush/FUA: distinguish durable writes from cache visibility, with bounded NFSv4 delegation comparisons.
Updated
NVMe over Fabrics: transports, discovery and security boundaries
Distinguish NVMe/TCP, NVMe/RDMA and NVMe/FC from discovery, path selection and security, without treating a transport label as a performance guarantee.
Updated
O_DIRECT alignment: query the file, not a fixed block size
Query O_DIRECT per-file alignment and distinguish cache hints, prefetch, NOWAIT and mincore residency from actual I/O and durable writes.
Updated
Phi failure detectors: suspicion is not a failure probability
What a phi score measures, why Cassandra's score differs from the original model, and why suspecting a replica does not fence it. Includes reproducible synthetic examples.
Updated
RocksDB: a WAL sync is not a memtable flush
RocksDB WAL sync, memtable flush, write stalls, read snapshots, checkpoints and backups have distinct recovery and retention contracts.
Updated
Disk scrubbing: detection time is not a durability guarantee
How scrub coverage and irregular inspection intervals limit detection evidence, with exact reproducible examples of waiting-time bias.
Updated
SIEVE cache eviction: one bit is not scan resistance
Explain SIEVE versus CLOCK and LRU, replay cache traces and FIFO capacity anomalies, and separate miss counts from storage performance claims.
Updated
Sleeping disks: one readable block is not a recoverable array
An exhaustive check of two historical XOR codes separates serving a read without spin-up from recovering every data symbol. Includes code, all 512 availability states and explicit limits.
Updated
Snapshots and backups: which failures can you recover from?
Separate snapshot consistency, independent recovery copies and RPO/RTO targets, with source-backed OpenZFS, PostgreSQL and VSS examples.
Updated
SSD wear indicators: estimates, failure evidence and NVMe scope
Percentage Used estimates consumed endurance; it is not a failure probability. Available Spare measures a different quantity, and a read-only warning is not a promise of continued data access. Historical SATA failures and Google's field study illustrate these limits, but neither calibrates modern NVMe warning coverage. Keep device identity, log identifier and counter scope with every observation.
Updated
Slow RAID hours are not request tail latency
Slow-drive hours and request percentiles count different events. Source review and exact examples distinguish fan-out, hedged reads, cancellation work and replica dependence.
Updated
Unnamed files: publishing is not replacing
A native Linux observation separates O_TMPFILE staging, linkat publication, no-replace rename and atomic exchange from crash durability.
Updated
WAL, checkpoints and torn pages: three different recovery guarantees
WAL recovery boundaries: PostgreSQL hint-bit writes, page-image compression and statistics traps, plus InnoDB doublewrite and SQLite commit semantics.
Updated
B-epsilon trees: visible updates, deferred work
How buffered tree updates remain query-visible, why blind upserts differ from read-modify-write, and why equal flush counts can hide different completion times.
Updated
Missing SMART ages and the limits of drive-age comparisons
A full Q4 2025 raw-data audit finds 18 TB drives in a mislabeled capacity group and shows why excluding 106 missing-age drive-days also excludes 19 failures.
Updated
Content-defined chunking: boundaries are not identity
Reproducible FastCDC examples show how file framing, parameter changes and terminal chunks alter measured reuse, even when file contents remain unchanged.
Updated
Crash-consistency tests: the oracle is part of the result
What ALICE, CrashMonkey/ACE, SQLite and Ananke actually test: fault boundaries, separate recovery outcomes and an executable correctness-oracle example.
Updated
Datacenter failure tickets are not failed-device counts
A hardware ticket, a failed device and a service outage are different counting units. Review the reported fleet figures, separate response time from repair time, and reproduce why a 120-second failure-burst rule need not limit a burst to 120 seconds.
Updated
Fast file clones: where the copying cost goes
Separate reflinks, hard links and in-kernel copies; trace deferred costs, Btrfs maintenance, shared-space accounting and FAST 2020's research limits.
Updated
CMR is not refresh-free: adjacent-track interference in HDDs
Repeated HDD writes can trigger protective refreshes even with CMR. Separate write-induced ATI, read-channel interference and SMR band rewrites before diagnosing latency.
Updated
HDD workload ratings: count reads, check the clock
An HDD workload rating is not SSD TBW or a failure countdown. Power-on hours, read traffic and maintenance scans change what the annualized number means.
Updated
HDD write verification: completion, persistence and comparison
Read-after-write verification, IRAW and SCSI VERIFY answer different questions from flush or FUA. Learn what a successful check does and does not establish.
Updated
LMDB readers: a stable snapshot can exhaust a fixed map
A controlled LMDB experiment separates reader lifetime, reusable pages and map capacity: why a live reader can stop writes without holding the writer mutex.
Updated
Why deleted NFS files become .nfs files
Silly rename, open-file preservation and a misleading Linux comment: trace NFS deletion from the 1985 paper to a pinned implementation and RFC erratum.
Updated
NVMe multipath: ANA access states and Linux path policies
Separate NVMe namespace identity from ANA access states: NSID scope, identifier lifetime, Linux path policies and controller-level queue accounting.
Updated
Object ETags: checksums and conditional writes are different
S3 ETags, full-object and composite checksums, version IDs and write preconditions: what each validates, and how to avoid checking download metadata instead of bytes.
Updated
RAIDZ expansion: why old blocks keep their parity cost
Reflow spreads old blocks across more disks without changing their data-to-parity ratio. A reproducible model separates retained parity overhead, reported space and snapshot-aware rewriting.
Updated
Replica repair: what has actually converged?
Replica repair coverage, quorum overlap and set reconciliation: why matching keys, successful responses and compact summaries do not prove identical durable state.
Updated
Replication acknowledgements: received, durable, applied
Compare replication acknowledgements, WAL retention and logical failover readiness; distinguish durable writes, replay and missing or unusable slots.
Updated
Sparse files, preallocation and thin-pool capacity
File size, allocated blocks and thin-pool headroom are different measurements. Learn what preallocation, hole punching and zero-filled reads actually establish.
Updated
SQLite WAL checkpoint starvation: success is not completion
Measured SQLite WAL checkpoints and stale-snapshot retries: why restarting a transaction can still overwrite an update when the application reuses old values.
Updated
Correlated SSD failures: what the Alibaba measurements count
Han et al. (FAST 2021) report that 12.9% of failures were intra-node and 18.3% intra-rack at a 30-minute grouping threshold. These are shares of recorded failures, not percentages of all installed drives. Reading the analysis code also separates a group-size statistic from a prediction of the next drive failure.
Updated
SSD mapping caches: why DRAM size is not a speed rating
Separate SSD maps from data buffers, explain cache admission and HMB, and reproduce memory, scan and request-versus-byte hit examples.
Updated
SSD firmware-hour failures: model, revision and runtime matter
A healthy wear indicator does not rule out a firmware-hour defect. Vendor advisories describe separate 32,768-, 40,000- and 28,224-hour issues with different affected models and recovery outcomes. Match the exact drive and firmware, then check fresh runtime evidence.
Updated
SSD thermal throttling: temperature and NVMe counters
A hot SSD reading, a thermal-management event and a drive failure are different observations. Historical fleet studies show platform-dependent temperature associations, not a universal failure threshold. NVMe exposes distinct temperature and action counters; their units, capabilities and specification revision determine what a reading can establish.
Updated
Storage fencing: why a lost heartbeat does not stop writes
Delayed I/O, SCSI and NVMe reservations, host identity and fencing tokens explain why path failover alone cannot protect shared storage.
Updated
Storage timeouts: an expired timer is not a cancelled write
Distinguish drive recovery limits, CDL descriptors, host timeouts and multipath queueing; preserve retry identity when write completion is uncertain.
Updated
Tape shoe-shining: how much a backhitch costs, what speed matching removes, and what the logs do not show
Why tape backhitches interrupt streaming, when filemarks confirm buffered writes, and how host supply and explicit byte units change a buffer-drain estimate.
Updated
XFS delayed logging: which commit is durable?
Separate XFS commit from persistence, relogging from deferred recovery, and journal atomicity from online visibility.
Updated
ZNS resource limits: why closing a zone may not unblock writes
Open and active zone budgets, Linux write plugging, native versus emulated Zone Append, and the limits of ZNS performance comparisons.
Updated
Btrfs tree logs: why a read-only mount can still replay writes
Separate Btrfs fsync logging, transaction commit, read-only mount recovery and destructive log clearing, with a reproducible Linux 6.15/6.16 option audit.
Updated
Why a valid checksum can still return the wrong data
Checksums, identity, freshness, Linux verity and disk encryption: what successful verification does and does not prove, with reproducible examples.
Updated
F2FS: when a successful fsync is not a recovery promise
A pinned Linux source audit separates F2FS checkpoints, roll-forward recovery and the fsync success path when checkpointing is disabled.
Updated
Perseus: why slow drives are not always broken drives
Perseus detects drive slowdowns, but shared-path faults can escape its peer model. Review host scheduling, benchmark limits and checked ratio examples.
Updated
HDD mechanical faults: evidence and diagnostic limits
Shock models, vibration tests and vendor counters reveal different parts of HDD mechanics. They do not provide a universal damage threshold or a failure forecast.
Updated
NAND erase blocks: geometry is not write amplification
More pages per erase block can increase work per collection, but the live-data fraction, mapping policy and workload determine the write cost. Separate historical chip parameters, simulated SSD results and a reproducible accounting example.
Updated
NVMe health log: warning bits, wear estimates and counter limits
Read NVMe health counters with the right units and version: wear is not failure probability, host writes are not NAND writes, and warnings are not a fault history.
Updated
Optical disc lifetime: error thresholds, percentiles and uncertainty
Optical-disc lifetime is not a warranty: CD/DVD error units, survival estimates, Blu-ray HTL/LTH, and a source-level check of M-DISC longevity claims.
Updated
RAID rebuild risk: read-error models, field evidence and resilver boundaries
A datasheet error-rate limit is not a measured rebuild-failure probability. Under an illustrative independent-error model, reading five 12 TB survivors at 1 error per 10^14 bits gives 99.2% for at least one read error; five 20 TB survivors at 1 per 10^15 bits give 55.1%. Those are different read volumes, 60 TB and 100 TB, and neither result estimates the chance of losing an entire array. Rebuild, resilver, spare activation and checksum verification are separate recovery milestones.
Updated
Reed-Solomon recovery needs more than a fragment count
Exact GF(7) examples separate MDS recovery, missing fragments and silent corruption, showing why even a unique decoded answer can be wrong.
Updated
Eleven nines: what a durability model does not prove
A durability target is not a measured loss rate. Recomputing a historical 20-shard example gives about 1.06e-11 per year with a Poisson event model and 7.72e-12 with a distinct-shard binomial model. The difference is a modeling choice, not evidence that one cloud provider is safer.
Updated
SATA and SAS link errors: what counters can and cannot diagnose
SATA error flags and SAS PHY counters identify observations in the I/O path, not the component that must be replaced. Preserve endpoint identity, timestamps and reset history before comparing counts. Historical field studies show why a disk-only view is incomplete, but their event shares are not modern SAS failure rates.
Updated
Sector-error prediction: what adaptive scrubbing gains
A 2017 simulation shortened error-detection time. That is not a measured reduction in data loss, and its accelerated-scrub budget is not the false-positive rate.
Updated
What smartctl and smartd can and cannot tell you about a failing drive
A successful smartctl process is not a complete health measurement. ATA exit bits distinguish command problems, threshold warnings and logged errors; PASSED is a reported check result, not a guarantee against failure. This guide separates those signals, explains smartd alert prerequisites and preserves unknown readings in tested collector examples.
Updated
SMR stalls, cache limits and the errors SMART can miss
An SMR performance stall and a failed write are different observations. Research explains how internal cleaning can delay work; historical WD40EFAX reports document command errors despite a passing SMART health result. Neither establishes that cache exhaustion caused every reported error, nor that SMR has a higher fleet failure rate.
Updated
Soft updates: consistent metadata without a full journal
Why write ordering can leave reclaimable space instead of unsafe pointers, and why background fsck and metadata consistency do not prove application durability.
Updated
SSD sanitization: why TRIM and a successful command are not enough
File deletion, deallocation and sanitization establish different things. A FAST study, NIST SP 800-88r2 and NVMe status semantics explain what evidence to collect before an SSD leaves your control.
Updated
What LTO TapeAlert can tell you about a failing tape, and what it cannot
TapeAlert reports conditions, not a cartridge failure rate. Flag meanings depend on whether the target is a drive or a library, and reading the log can clear the evidence. LTO error-rate specifications describe a separate model: they do not establish annual data durability or the health of unread data.
Updated
USB storage resets: recovery is not a root-cause diagnosis
A UAS abort or USB reset identifies a recovery action, not the failed component. Pinned Linux code, a historical incident and SMART documentation separate transport symptoms, compatibility quirks and disk-health evidence.
Updated
Filesystem aging: the state SSD warm-up does not describe
Separate filesystem layout history, free-space fragmentation and SSD conditioning, with three USENIX studies and a reproducible allocation counterexample.
Updated
ZFS and Btrfs checksum counters: coverage and blind spots
Checksum counters expose corrupt reads that device health can miss, but zero errors do not prove integrity or identify which component failed.
Updated
Linux atomic writes: untorn does not mean durable
Atomic writes, durable completion and powersafe overwrite protect different boundaries. Check Linux request geometry and SQLite's outside-range assumption.
Updated
NAND read-retry: when a successful SSD read takes longer
Why retry counts do not determine SSD read latency: chip evidence, pipelined recovery and reproducible synthetic stage-timing examples.
Updated
SMART replacement prediction: recall, F-score and random splits
Botezatu et al. report 98% Seagate recall; their Hitachi table gives 79%, while the prose says 81%. These random-split results are not a prospective deployment estimate.
Updated
SMB handles and leases: reconnect is not write durability
Durable and persistent SMB handles, caching leases and write-through protect different boundaries. Learn what to verify after a disconnect or failover.
Updated
SSD read reclaim: why read-only workloads can cause writes
Separate read disturb, read retry and data relocation; compare device measurements with simulations, and handle zero-host-write intervals without inventing a WAF.
Updated
SSD steady state is not a timer
Why SSD warm-up duration is not proof of steady state: source review, two independent stability checks and six reproducible synthetic counterexamples.
Updated
ZFS ZIL and SLOG: the boundary between logging and persistence
ZIL, SLOG and transaction groups protect different boundaries. Pinned OpenZFS code explains indirect records, sync settings and misleading log-byte counters.
Updated
Fail-slow NVMe SSDs: what SMART and heartbeats miss
Separate fleet counts from model averages, review uncertain follow-up dates, and test how a live heartbeat can miss stalled storage work.
Updated
Parity declustering: less load per disk, not fewer reads
An executable eight-disk example separates repair-read volume, survivor load and stripe width, with evidence from ASPLOS, Panasas and OpenZFS dRAID.
Updated
Persistent memory: DAX is not a durable commit
ADR, eADR and MAP_SYNC protect different boundaries. A source-linked review and executable crash-state example separate addressability, visibility, persistence and atomicity.
Updated
Media error or link error? What SCSI sense codes and Linux ATA log lines can and cannot tell apart
ATA registers, Linux's error mask and SCSI sense data describe different layers. They can distinguish a reported media error from a transfer error, but do not uniquely identify the failed component. Recovery thresholds produce conditional verdicts, not guaranteed actions: the audited libata path does not force a SATA ATA disk into PIO. A reproducible comparison also shows why decoding must be tied to a kernel revision.
Updated
38% of failed datacenter SSDs showed none of the four SMART symptoms; the ones that did failed 3 to 20 times as often
In Narayanan et al. (SYSTOR 2016), a study of over half a million SSDs in Microsoft datacenters over nearly 3 years, 62% of devices that fail-stopped had shown at least one of four SMART symptoms (uncorrectable or CRC data errors, sector reallocations, program or erase failures, and SATA downshifts) and 38% had shown none. Devices with a symptom failed 3 to 20 times as often as devices without one, with data errors the strongest at up to 20 times. Of the models whose failure rate could be compared with vendor figures, two consumer models exceeded the published 0.61% to 0.73% annual failure rate. A multi-factor classifier reached 87% precision and 71% recall, but it was scored with 5-fold cross-validation on one operator's data. Maneas et al. (FAST 2020) and Backblaze (2016) show the same pattern in other fleets: symptoms raise risk and a large share of failures give no warning.
Updated
What an unexpected power cut does to SSDs, and why power-loss protection is hard to check from outside
Historical power-fault tests exposed SSD data and ordering failures, including on devices labelled with PLP. Those outcomes do not identify a failed capacitor or establish a current-product failure rate.
Updated
Google measured a disk MTTF of 10 to 50 years but a node MTTF of 4.3 months; 37% of node failures came in bursts
In Ford et al. (OSDI 2010), one year of data from tens of Google storage cells of 1000 to 7000 nodes each, the mean time to failure was 10 to 50 years for a disk, 10.2 years for a rack, and 4.3 months for a node, so node failures, not disk failures, drove availability. 37% of failures were part of a burst of at least 2 nodes. In the authors' model, a 10% cut in the disk failure rate raised stripe availability by less than 1.5%, while a 10% cut in the node failure rate raised data availability by 18%. Maneas et al. (FAST 2020, NetApp SSDs) and Schroeder and Gibson (FAST 2007, HPC1 disks) also found strong clustering of drive replacements in time, so a failure rate alone does not describe risk.
Updated
Storage consistency: ordering is not a transaction or a durability promise
Linearizability, transactions, four session guarantees and compare-and-swap: identify the scope of a storage guarantee before relying on it for correctness.
Updated
Storage latency: when the benchmark stops asking
A completion-driven load generator can hide queueing during a stall. A reproducible synthetic example explains coordinated omission, timing boundaries and why p99 values cannot be averaged.
Updated
Key-value separation: the cleaning bill moves elsewhere
WiscKey, HashKV and BlobDB separate value movement from key sorting. A reproducible byte model shows when value-log cleaning can erase the saving.
Updated
LDPC in NAND: the read cost of confidence
How soft information is acquired, why decoder structure changes the result, and which error-rate denominators an SSD reliability claim must disclose.
Updated
LSM tombstones: deleted is not yet reclaimed
Separate delete visibility, crash durability and space reclamation: snapshots, Lethe's delete-aware compaction and RocksDB's range-tombstone safeguards.
Updated
Disk replacement rates are not disk lifetimes
Field replacement logs count operational decisions, not verified component lifetimes. Drive age, fleet event gaps and repeated-event rates answer different reliability questions.
Updated
BCH, LDPC and the RBER aggregation trap
A drive-wide raw error rate cannot reveal the hardest codeword. How to compare BCH and LDPC evidence without confusing fleet telemetry with decoder limits.
Updated
NVMe FDP: placement handles are not physical addresses
How reclaim units, handles and placement identifiers fit together, why FDP differs from ZNS, and what a credible write-amplification comparison must measure.
Updated
Optical disc rot: what public ageing studies measured, which error counters (BLER, PI sum 8) warn first, and what they cannot predict
Recordable discs fail mainly because the organic dye layer breaks down or, on discs with a silver reflective layer, the metal corrodes, and the public tell-tale is a rising correctable-error counter before data is lost: block error rate (BLER, counted at the first CD decoder) against a specified maximum of 220 per second on CD, and parity-inner errors summed over eight blocks (PI sum 8) against a maximum of 280 on DVD. In the NIST and Library of Congress study of discs bought in 2005, all 7 CD-R products reached an estimated life of more than 30 years at 25 C and 50% relative humidity (5 of 7 more than 45 years), while the 15 DVD products spread from under 15 years (2 products) to more than 45 years (7 products). Canada's conservation institute ranks a gold-layer phthalocyanine CD-R above 100 years and a dye or non-dye single-layer Blu-ray recordable at 5 to 10 years, and says discs recorded with poor technique can fail in as little as 2 to 10 years. These are accelerated-ageing estimates of temperature and humidity effects only, for disc products of 2005 or earlier, so they rank media and set expectations but are not a failure rate for any disc in use today.
Updated
Two clocks for one GitLab.com loss: 00:00 UTC and 23:30 UTC, at least 5,000 projects in the postmortem, 5,037 in the log
On GitLab.com’s one primary and one hot-standby secondary, the 10 February 2017 postmortem prints two ends for database rows lost on 31 January 2017: the introduction says 17:20 and 00:00 UTC on that date, and the data-loss section says 17:20–23:30 UTC. It says the exact loss is hard to estimate: at least 5,000 projects, 5,000 comments, and roughly 700 users. The incident log archived at 2017-02-02 03:47:19 UTC prints ±6 hours, 5,037 projects, ±4,979 comments, and 707 users that were hard to confirm from Kibana. Those figures are vendor claims, not a census of objects that stayed gone. The same postmortem says projects restored from remaining Git data came back only in basic form, without issues or merge requests.
Updated
512e, 4Kn and alignment: geometry is not durability
Separate logical and physical sectors, partition starts and individual writes. Four reproducible examples show why aligned partitions do not guarantee whole-sector I/O.
Updated
SSD over-provisioning: the denominator changes the answer
Convert SSD reserve percentages correctly, reproduce a uniform-write amplification model, and see why cleaning policy and workload limit its predictions.
Updated
Deduplication ratio does not predict restore cost
A cold-cache trace separates container rereads from unused fetched bytes, with six reproducible examples and a review of two USENIX papers.
Updated
Discard is not zeroing: what TRIM, UNMAP and fstrim actually promise
A discard request, zero-filled readback and reclaimed physical space are different observations. Protocol and Linux documentation show how to interpret each without false assurances.
Updated
In an 18-month Berkeley study of 368 disks, 7 data disks failed against 13 enclosure backplane failures; disks were 90% of parts but about 4% of errors
In Talagala and Patterson (UC Berkeley, 1999), 18 months of operation of a prototype with 20 PCs and 368 SCSI data disks, 34 components were replaced. Only 7 of the 368 data disks (1.9%) failed, the lowest percentage of any failing component, against 13 of 46 disk enclosures (28.3%, backplane faults) and 6 of 24 IDE system disks (25.0%). Disks were about 90% of the components but only around 4% of 688 logged error instances, while SCSI timeouts and parity errors were 49% (87% without network errors), and no restart was caused by a data disk. Ford et al. (OSDI 2010) later found a disk MTTF of 10 to 50 years against 4.3 months for a node, and Schroeder and Gibson (FAST 2007) found disks were 20% to 50% of hardware replacements, a count not a rate.
Updated
Local repair is not whole-node repair
A reproducible LRC comparison separates data-fragment reads, parity repair and whole-node accounting without equating read savings with durability.
Updated
NAND copyback: faster relocation is not fewer writes
Why on-chip page relocation can bypass controller ECC, why destination placement matters, and why lower NAND-channel traffic is not lower write amplification.
Updated
NAND paired pages: why a later write can damage earlier data
Two NAND pages can share cells. Distinguish interrupted programming, neighbor interference and the protection promised by an SSD.
Updated
RAID write hole: why a bitmap is not a write journal
A bitmap tracks dirty regions; PPL repairs a narrower parity hazard; a journal preserves replay data. A reproducible XOR example separates these guarantees.
Updated
RAID write penalty is not a speed ratio
Read-modify-write, reconstruct-write and full-stripe accounting: why four or six member operations do not mean one-quarter or one-sixth of full-stripe speed.
Updated
SMART zeros, missing attributes and failure-day selection
A complete Q4 2025 raw-data audit separates 66 failed drives with five measured zeros from 170 with partial zero-only readings, and explains why complete-case filtering changes the manufacturer mix.
Updated
What do public storage-incident postmortems show about drives that fail together?
Four public postmortems show drives failing together or in bulk because they shared something (a controller, a lot, a firmware), and the reports separate what the logs showed from what the operator only suspects. InfiniCLOUD (October 2026): 6 of 11 disks in a ZFS RAIDZ3 pool sat on one controller; one disk failed at 01:43 JST on 1 October, and a fourth had failed by 19:21 JST on 2 October, 41 h 38 min later, beyond the 3-disk tolerance; the cause of the last three failures is still unknown. Echelon VPS (February 2026): both NVMe drives of one mirror, same manufacturing lot and nearly equal write volume, hit a firmware assertion 4 min apart and took 218 instances offline for 94 min. GitHub (January 2016): a 2 h 6 min outage after a power blip, when over 25% of servers could not see their own drives after rebooting, a known firmware issue. Seagate (January 2009): a firmware event-log bug hung drives at power-up when the log counter was at entry 320 or 320 + n x 256, and the first fix was withdrawn after it disabled some 500 GB drives. None of the four reports publishes the raw logs, and three of the four name no drive or firmware version.
Updated
Azure disk-error prediction caught 36.5% of faulty disks at a 0.1% false-positive rate, and random cross-validation said 91.6%
In Xu et al. (USENIX ATC 2018), Microsoft's disk-error predictor, CDEF, found 36.50% of faulty disks on one test set when it was allowed to wrongly flag only 0.1% of healthy disks (29.67% to 41.09% across three test sets), with about 10,000 healthy disks per 3 faulty ones. Scored with a random split, the same task gave 91.64%, which is why the authors insist on testing forward in time. Adding system-level signals such as OS events to SMART raised the average catch rate at that false-positive rate from 27.6% to 30.3%, and to 35.8% with feature selection. In production at Azure the authors report about 63,000 fewer minutes of VM downtime a month. The data are one cloud system in one month, and labels come from engineers' root-cause analysis of service problems.
Updated
Enterprise SSD replacements ran at 0.22% a year across 1.4 million NetApp drives, with firmware the biggest swing
In Maneas et al. (FAST 2020), about 1.4 million SSDs in NetApp enterprise storage systems had an average annual replacement rate (ARR) of 0.22%, with models ranging from 0.07% to nearly 1.2%. That is well below the 4% to 10% of Google's flash drives removed within four years and the 2% to 9% a year the same paper quotes for hard disks, though the definitions of replacement differ. Within the NetApp data, the earliest firmware versions of some models were replaced up to 8 to more than 10 times as often as later ones, and the chance of a second replacement in the same RAID group within a week was 9.39% against 0.0504% in a random week. One third of replacements were predictive and another third were severe SCSI errors. Method limits are on the note.
Updated
Network is the largest fail-slow hardware type: 29 of 112 occurrences in 101 reports
Network is the largest hardware type in Table 3 of Gunawi et al., FAST 2018: 29 of 112 root-cause occurrences, ahead of CPU 25, disk 23, SSD 22, and memory 13. Those 112 occurrences come from 101 fail-slow reports at 12 institutions, incidents reported from 2000 through 2017, and a report can name more than one cause. Device errors account for 40 occurrences and firmware issues for 20. Temperature, power, environment, and configuration sum to 44 of 112 (39.3%), which rounds to the proceedings’ statement that 39% of root causes are external factors, not 39% of the 101 reports.
Updated
Replacing disks at 200 reallocated sectors removed 88% of triple-disk RAID failures at one vendor
In Ma et al. (FAST 2015), about 1 million SATA disks in backup systems from one storage vendor showed that the count of reallocated sectors (RS) correlates strongly with whole-disk failure. Replacing a disk once its count passed 200 eliminated about 88% of the RAID failures caused by three-disk errors, which were 80% of all RAID failures, or about 70% of all disk-related incidents. The same paper says the signal is incomplete: in simulation, a threshold below 200 caught 52% to 70% of impending whole-disk failures, with 0.8% to 4.5% false positives. Google's FAST 2007 data and Backblaze's 2016 SMART tally point the same way on reallocated sectors, and both also show failed drives with no warning.
Updated
0.86% of nearline disks and 0.065% of enterprise disks developed a checksum mismatch in 41 months
In Bairavasundaram et al. (FAST 2008), 3,088 of about 358,000 nearline disks (0.86%) and 767 of 1.17 million enterprise-class disks (0.065%) developed at least one silent checksum mismatch over 41 months, in field logs covering 1.53 million disks. That is about 400,000 mismatched blocks in all, but the burden is lopsided: the median affected disk had 3 mismatches, the mean was 104, and the worst disk had 33,000. The paper's rates are per disk and come from one vendor's support logs, so they do not say how often a file is corrupted. An independent 2007 CERN report found 22 mismatching files among 33,700 checked, about 1 in 1,500, using a different method and a different population.
Updated
34.4% of RASR failures are not the SSD: 20.5% human mistakes, 13.9% cables, 31.2% labeled Failed Device
In Xu et al., USENIX ATC 2019, 34.4% of Reported As “SSD-Related” (RASR) failures are not caused by the SSD device. That 34.4% is human mistakes (20.5%) plus faulty cables (13.9%), inferred from which repair worked. It is not the complement of a measured device failure: Table 11 labels 31.2% Failed Device because replacing the SSD is the last resort, 11.9% Transient, and 22.5% Undetermined (FSCK 16.5% plus Data Check 6.0%). The paper prints 20.5 and 34.4. It does not print 22.5 or the addition 20.5 + 13.9.
Updated
What fails most often: unreadable data on disks, uncorrectable reads on SSDs, and bus errors on the link
The most frequent field fault is not a dead drive but data that cannot be read: 3.45% of 1.53 million hard disks developed at least one latent sector error in 32 months, and 20.3–90.5% of Google's flash drives (ten models, first 4 years; 3 for eMLC) had at least one uncorrectable error and 19.1–62.7% a final read error, while 4.2–34.1% of Facebook's SSD platforms reported at least one uncorrectable error. Link faults show up separately as CRC counts (about 2% of Google's disks), and the public counters miss a large share of failed drives: over 56% of failed drives in the Google study and 23.3% in Backblaze's 2016 log showed no signal on the SMART attributes each used.
Updated
Over 56% of failed drives had no count on the four strong SMART signals
Over 56% of failed drives in the Google production population of more than 100,000 consumer-grade serial and parallel ATA drives (December 2005–August 2006) had no count in any of the four strong SMART signals (scan errors, reallocation count, offline reallocation, and probational count), so models based only on those signals can never predict more than half of the failed drives.