Google measured a disk MTTF of 10 to 50 years but a node MTTF of 4.3 months; 37% of node failures came in bursts

The answer. In Ford et al., the unit that fails most often is the node, not the disk. The mean time to failure in Google's cells was 10 to 50 years for a disk, 10.2 years for a rack of nodes, and 4.3 months for a node, and the node count includes software restarts and other transient causes. Because nodes fail so much more often, the authors find they matter far more for data availability, even though disk failures are the ones that are permanent. In their model with 3-way replication, a 10% cut in the disk failure rate raised stripe availability by less than 1.5%, a 10% cut in the latent disk error rate had a negligible effect, and a 10% cut in the node failure rate raised availability by 18%.

Failures arrive in bursts. When the authors group node failures that occur within 120 seconds of each other, 37% of failures are part of a burst of at least 2 nodes, and they estimate that close to 37% are truly correlated, since only 8.0% of non-correlated failures could be merged by chance. Most large bursts are tied to rack-level or multi-rack events such as shared switches, power, or planned kernel upgrades. Ignoring such correlation makes a model too optimistic: with correlated failures included, even a 90% reduction in recovery time gave only a 6% reduction in unavailability.

Disk-level numbers are in line with other studies. The authors say their disk figures are comparable with published disk and storage subsystem failure rates, and background scrubbing found checksum mismatches in 1 in 10^6 to 10^7 older data blocks, concentrated on a small number of disks. Disk errors are therefore a small contributor to unavailability but a real one for data integrity, which the system handles with scrubbing and verification on reads.

Other fleets show the same clustering. Maneas et al. found a 0.0504% chance of a drive replacement in a random week but 9.39% in the week after a previous replacement in the same RAID group, more than 180 times higher, and 52% of consecutive replacements fell within a week. Schroeder and Gibson found a correlation coefficient of 0.72 between disk replacement counts in consecutive weeks of the HPC1 system, and the expected number of replacements in a week varied by a factor of 9 depending on how busy the previous week was. The three fleets, devices, and measures differ, so the sizes are not comparable, but all three reject the assumption of independent failures.