What do public storage-incident postmortems show about drives that fail together?

The clearest pattern is that redundancy was counted in disks but the failure domain was bigger than a disk. InfiniCLOUD's report says six of its eleven RAIDZ3 disks hung off one controller, so a controller problem could take out more disks than three-way parity tolerates. Echelon's report says its mirror halves came from the same lot with almost the same write volume and hit the same firmware assertion four minutes apart. GitHub's servers shared a hardware class and a firmware issue, and clusters lost all members although the members were spread over different racks. In each case the reports describe a shared cause, not a coincidence of independent faults.

The reports are also clear about the limits of early signals. InfiniCLOUD's monitoring raised an alert 10 minutes after the first symptom, but no data read errors were logged then and the service was judged unaffected; the unreadable system files were found 4 h 53 min after the first symptom. ZFS recorded errors against all ten remaining disks as DEGRADED because it could not tell which disk held the data it could not rebuild, which the report says does not mean ten disks had failed. A pool state is therefore a statement about the pool, not a per-disk diagnosis.

Firmware shows up as a trigger or a spreader in all four, but the evidence differs. Seagate's notice gives a precise mechanism (event-log entry 320 or 320 + n x 256 at power-up, plus a factory test fill pattern) and a date after which new drives were not affected (12 January 2009). GitHub names only 'a known firmware issue' and a cold power-drain that made the disks visible again. Echelon says its affected firmware revision was replaced fleet-wide within eleven days on 214 racks. InfiniCLOUD says it is still investigating whether an older controller firmware's error recovery, such as aborting commands or resetting a disk, spread the problem. Only Seagate's mechanism is documented to the level of a counter value.

What remains unknown is as important as the timelines. No report publishes raw SMART, NVMe health log, SCSI sense or kernel log lines, so this page cannot say which public counter would have warned first. Three of the four name no drive or firmware version. InfiniCLOUD's own report says the cause of the three disk failures on 2 October is still uncertain and could lie in the controller or in the disks. Seagate's fix itself failed: The Register reports that update SD1A was withdrawn after it disabled some 500 GB drives, which is a reminder that a firmware remedy is a change with its own failure modes.