Fibre Channel

Follow an I/O through frames, sequences and exchanges, and distinguish link-buffer availability from command completion.

Glossary entry on hesela.dev · JSON record · Frame · Sequence · Exchange · Buffer-to-buffer credits · Buffer-cycle deadlock · NVMe fabrics and transport choices

On this page7 sections

In one minute

Fibre Channel is a frame-based networking architecture used to carry storage protocols between node ports, directly or through a fabric.

Start with the operation, not the cable. An exchange organizes related sequences; a sequence contains frames. In a trace, preserve port addresses, OX_ID/RX_ID, SEQ_ID and SEQ_CNT together. Looking at an exchange number without its endpoints and lifetime can join unrelated work. [6]

FC-NVMe reuses elements of FCP transport while carrying different command semantics. Its I/O can reuse FC-2 TYPE 08h, so identifying the upper-layer operation requires protocol context, not just the frame TYPE. A header constant alone is not a reliable SCSI-versus-NVMe classifier. [6]

Size has a boundary too. The base header is 24 octets and the data field can reach 2112 octets, including optional headers. Negotiated limits can be smaller. These are not 2112 guaranteed application bytes. [2] Linux's FC_MAX_FRAME is 2136 bytes, counting the header and data field but not the CRC or delimiters. [3] RFC 3643 includes the CRC in frame content and represents delimiters separately for IP encapsulation; its encoding is not a universal native-link format. [7]

Buffer-to-buffer credits belong to each direction of a link. Sending a frame spends one credit, regardless of frame length. In the basic R_RDY mode, freeing a receive buffer returns permission to send another frame. At zero credits the sender waits; this does not by itself prove a failed disk or lost data. [4]

R_RDY is not a SCSI status response. Linux represents command status in FCP_RSP separately from link flow control. Consequently, a credit return cannot demonstrate command success or durable media state. This is a protocol-layer inference, not a power-failure experiment. [5]

When transmission stalls, record direction, the observation interval, credit availability and error counters alongside command latency. Cisco documents insufficient credit for link speed/distance and credit loss after damaged frames or R_RDY primitives. Diagnose these before treating every pause as a slow storage device; credit-return variants such as ER_RDY also exist. On Cisco MDS E ports, the cited BB_SC_N credit-recovery feature supports R_RDY, not ER_RDY mode. Do not assume a recovery mechanism applies across modes. [4]

Preventing buffer overflow does not ensure forward progress. Cherkasova, Kotov and Rokicki studied cyclic buffer dependencies in 1996: frames can wait on one another through a closed chain of occupied buffers. Their routing analysis separates this from raw link capacity. It is a historical model, not a measurement of modern switches. [1]

In a switched fabric, FLOGI establishes fabric parameters and obtains a fabric address; PLOGI establishes peer-port parameters. [2] FLOGI does not make a switch a universal prerequisite: Linux libfc also handles it in direct point-to-point operation. [8]

Sizing the credit window

A faster link can spend its credits sooner without receiving them back sooner. FCIA's 2024 interconnect presentation relates the required window to rate, round-trip delay and frame size. [9, slides 19-23]

For a fixed frame stream, let F be the intended frames per second and T the time from spending one credit until it can be reused. Covering that interval requires ceil(F * T) credits. With C credits, our idealized long-run sending fraction is min(1, C / (F * T)).

Worked examples, not switch sizing recommendations. Gb/s counts the assumed frame bits, not a marketed FC speed or application goodput.
Case	Gb/s	Bytes/frame	Return time (us)	Credits available / needed	Sending fraction
Reference	16	2000	100	64 / 100	64%
Half-sized frames	16	1000	100	64 / 200	32%
Twice the bit rate	32	2000	100	64 / 200	32%
Longer credit return	16	2000	150	64 / 150	42.7%
Larger window	16	2000	100	100 / 100	100%

This model assumes one direction, a continuously ready sender, fixed frame size and fixed return time. It omits framing and coding overhead, bursts, shared buffers, lost credits and downstream contention. T includes serialization and processing where relevant; propagation alone is a lower-bound approximation. Averages do not describe every busy interval.

Reproduce the table with the JavaScript module and compare its JSON assumptions and results. No hardware throughput was measured.

Check the arithmetic before copying a sizing rule

FCIA slide 22 uses 100 km, 2000-byte frames and 57.8 Gb/s. Its stated 2 x 10⁸ m/s follows c / n for n = 1.5, although the slide prints multiplication. Keeping its approximate inputs without intermediate rounding gives 3612.5 frames, rounded up to 3613 credits. Its 3636 result follows rounding the spatial frame length to 55 m first. These are arithmetic checks, not validated operating margins. [9]

Cisco's 9.x guide applies the 16-Gbps coefficient to its 1323-byte, 50-km example, then labels the conclusion 8 Gbps. Its own rule gives about 650 credits at 16 Gbps and 325 at 8 Gbps; literal upward rounding gives 651 and 326. The frame-size convention and approximation differ from FCIA's example. Do not mix either rule with an unexamined wire-rate or application-byte count. [4, Long-Distance ISLs]

A pause is not a deadlock diagnosis

Credit starvation means a ready sender is waiting for credit. A count of transitions to zero is not elapsed blocked time: preserve the counter definition, direction and measurement interval. Check credit-return delay and loss alongside downstream consumption. [4]

The 1996 paper's topology-throughput bound assumes uniform random traffic and infinite buffering. Its separate dependency-graph analysis concerns deadlock. Neither is a measured credit allocation for a present-day switch; a finite window calculation cannot establish deadlock freedom. [1, sections 2.1 and 3]

Compared to neighbors

Use link counters to investigate the path, command responses to investigate the operation, and the storage stack's persistence contract to evaluate durable writes.

Four observations, four different meanings
Observation	What it tells you	What remains to check
Frame CRC check	Integrity check over the covered frame content [7]	Address, generation and persistence of stored data
Sequence identifiers	How frames belong to a sequence [6]	Correct context and complete reassembly
Exchange identifiers	Association of related sequences [6]	Command response, status and recovery handling
Returned BB credit	A receive-buffer slot is available [4]	End-to-end command completion and storage durability

Uncertainty notes

Source-backed architectural explanation, not a fabric benchmark or a current product comparison.
The RFCs are dated architecture and encapsulation sources; Linux references are pinned to v6.12. Cisco behavior and configuration details are scoped to the cited MDS guide.

Sources

01Ludmila Cherkasova, Vadim Kotov and Tomas Rokicki. Fibre Channel Fabrics: Evaluation and Design. HICSS 1996, sections 2.1 and 3. DOI 10.1109/HICSS.1996.495447. Author-uploaded full text. · accessed 2026-10-19
02RFC 4338 (January 2006), sections 2.1-2.4: Fibre Channel architecture summary · accessed 2026-10-19
03Linux v6.12: fc_fs.h frame layout, identifiers and protocol types · accessed 2026-10-19
04Cisco MDS 9000 Interfaces Configuration Guide, Release 9.x: Buffer-to-Buffer Credits and Credit Recovery · accessed 2026-10-19
05Linux v6.12: fc_fcp.h, fcp_resp and SCSI status · accessed 2026-10-19
06FCIA: Inside a Modern Fibre Channel Architecture, Part 2 (2021), slides 7, 19, 23-31, 35 and 38 · accessed 2026-10-19
07RFC 3643 (December 2003), section 5: frame content and delimiter representation · accessed 2026-10-19
08Linux v6.12: libfc fc_lport.c, direct point-to-point FLOGI handling · accessed 2026-10-19
09FCIA: Fibre Channel Data Center Interconnects: 64G FC and More (18 July 2024), slides 19-23 · accessed 2026-10-19