When the cable, backplane or adapter fails instead of the disk: what link faults look like in logs and field data

In the largest opened field data, a disk-only view misses more than a quarter of subsystem failures. NetApp's physical interconnect failures were 27 to 68% of subsystem failures and disk failures 20 to 55%. In the low-end class the subsystem AFR was 4.6% against 0.9% for disks. The same disk models showed disk AFRs that varied little between systems (disk D-2: 0.6 to 0.77%) while their subsystem AFRs varied from 2.2 to 4.9%, so the difference sat in the shelf, adapter and cabling around the disk. This is a field result for 2004 to 2007 Fibre Channel and SATA shelves; it shows that link faults were a major source of failures then, not what share they are in a given system now.

Link faults are often counted as disk faults. The NetApp authors note that system administrators often replace a disk when it becomes unavailable, and that replacement-rate studies therefore end up close to the subsystem failure rate, not the disk failure rate. Schroeder and Gibson add that after a drive is blamed, tests decide whether it is replaced, that a vendor reported finding no problem in 43% of returned disks (a personal communication quoted in the paper), and that the hardware lines for cables and backplanes in the replacement logs are small: 2.2% for SCSI cables in COM2, 0.3% for SCSI backplanes in HPC1. Both can be true: a replacement log lists the part that was swapped, so a swapped disk that was never faulty is counted as a disk.

The log trail of a link failure is quick at the top and slow at the bottom. In the NetApp example the adapter timed out, reset after 14 seconds, found the device not responding after 36 seconds, ran out of paths after 46 seconds, and the RAID layer logged the disk as missing 166 seconds after the first event. The authors say a physical interconnect failure can be recovered by retries or tolerated by multipathing, so many never reach the RAID layer; what is seen there is only the part that was exposed. They also report strong clustering: after one failure in a shelf, a second of the same type was 10 to 25 times more likely than independence predicts for the non-disk types, because the disks share adapters, cables and terminators. For an operator this means that several disks vanishing from one shelf or one controller at nearly the same time is a link or shelf signature, not a coincidence of disks.

On a Linux host the useful discriminators are an ICRC flag, SError link bits, the Emask value and SMART attribute 199 reported together with no media errors, which the kernel documentation says often means a cable, connector or power problem. They cannot go further. The same bits can come from an inadequate power supply, a wrong transfer mode or an adapter, and no opened source says how often each cause occurs. Two caveats apply to the SMART side: attribute meanings are manufacturer-specific, and ID 183 means link downshifts on some drives and runtime bad blocks on others. A redundant path reduced interconnect failures by 50 to 60% in the NetApp data but did not remove them, because the authors name shelf backplane errors, which a second path does not cover, and a shared physical host adapter behind two logical ones, as likely reasons.