NVMe over Fabrics: transports, discovery and security boundaries

Three bindings, different requirements

InfiniBand, RoCE and iWARP are RDMA-provider examples. iWARP itself uses TCP/IP: comparing NVMe/TCP with NVMe/RDMA therefore compares NVMe bindings, not mutually exclusive underlying network protocols. [2, section 2]

An ordinary NVMe/TCP Admin or I/O connection maps to one queue pair, not an entire controller's collection of queues. TCP Transport 1.3 assigns discovery a default port of 8009 but defines no default TCP port for I/O controllers: use the reported transport service identifier. [1, sections 3.1.2 and 3.2]

Finding a subsystem is not accessing its data

A discovery controller returns connection information for NVM subsystems or other discovery services. It does not expose namespaces or implement I/O queues. Base 2.2 supports unique Discovery Service NQNs as well as the well-known discovery NQN. [4, section 3.1.3.3]

Revision 1.3 also defines a kickstart discovery connection carrying only KDReq/KDResp registration exchanges. Do not apply the ordinary queue-pair description to that special connection. [1, section 3.1]

nvme discover retrieves a discovery log; it is not equivalent to connecting to every advertised storage controller. Its v2.11 documentation describes TRTYPE, TRADDR, TRSVCID and SUBNQN as connection information. [6]

Linux native multipath selects eligible I/O paths and prefers ANA-optimized paths over non-optimized paths. [7] The operational inference is that a successful discovery response does not prove namespace access, alternate-path usability or a failover-time bound. Continue with namespace identity and ANA path selection.

Authentication and encryption are separate checks

SPDK documents TLS configuration separately from in-band authentication. Feature availability and combinations depend on the implementation, so a TCP transport label is not evidence that either has been enabled. [5]

Hesela's review checklist is to record the actual host/target versions, permitted identities, authentication policy and negotiated channel protection separately. Do not paste keys or secret material into an incident report. A subsystem NQN is a name, not a credential.

A digest is not an authentication tag

NVMe/TCP negotiates header and data digests independently during connection setup. Both use CRC32C; padding is outside their coverage. These are transport error checks, not proof of persistence on media. [1, sections 3.3 and 3.3.1.1]

RFC 3385 explains the security boundary: an attacker able to replace data can also replace its CRC. A valid digest therefore does not authenticate a peer or conceal data. [8] The v2.11 client exposes --hdr-digest, --data-digest and --tls separately. Record negotiated settings, not just requested flags. [9]

Keep transport digests separate from stored-data checksums, identity and freshness checks. Each protects a different boundary.

What a useful comparison must record

The following is a proposed measurement checklist, not a benchmark result:

• Host and target software, adapters, CPU/NUMA placement and network topology.
• I/O size, read/write mix, queue depth, concurrency and offered load.
• CPU consumption, throughput and latency distribution over the same observation window.
• Negotiated digests and security settings, errors, retries and any injected failure, with its recovery criterion.

ReFlex: why the client belongs in the result

Klimovic, Litz and Kozyrakis's 2017 ReFlex prototype used an IX dataplane with a custom remote-block interface over TCP, not the NVMe/TCP binding. Table 2 reports unloaded means for 4KB random reads at queue depth one on 10GbE. [10, sections 4.1, 5.1 and 5.2]

The familiar 21-microsecond difference applies to the IX client; the Linux-client difference is 39 microseconds. The separate 850K IOPS/core result used 1KB reads. Hardware-accelerated RDMA figures came from another presentation, not the same testbed. Section 4.1 also leaves barrier operations for future work. [10]

Hesela's interpretation: compare workload, client path and completion semantics before comparing transport names. These published means supply no confidence interval here; we did not reproduce the experiment. Download the transcribed values and provenance.

A transport name alone cannot settle which stack is faster. See latency sampling and coordinated omission before interpreting percentile charts.

Transport is only one layer

Architectural implication: an additional path is not an additional data copy, and network delivery is not an application durability contract. Use separate evidence for timed-out writes, writer fencing and flush/FUA persistence semantics.

Reviewed editions are explicit; this is not a claim that every implementation supports every feature. No hardware, failover, latency or durability experiment was performed. The FC reference is a historical mapping description, not a claim about the latest FC-NVMe edition.

• NVMe over Fabrics
• NVMe/TCP
• NVMe/RDMA
• NVMe/FC
• Discovery controller
• NVMe/TCP digest

Primary sources

1. NVM Express NVMe over TCP Transport 1.3, sections 2, 3.1, 3.2, 3.3.1.1 and 3.6
2. NVM Express NVMe over RDMA Transport 1.3, section 2
3. NVM Express NVMe over Fabrics 1.1a, section 7.2: Fibre Channel transport
4. NVM Express Base Specification 2.2, section 3.1.3.3: Discovery Controllers
5. SPDK NVMe over Fabrics target: transport support, TLS and in-band authentication
6. nvme-cli v2.11: nvme-discover
7. Linux native NVMe multipath: ANA states and path-selection policies
8. RFC 3385, section 10: CRCs do not protect against intentional alteration
9. nvme-cli v2.11: nvme-connect digest and TLS options
10. Klimovic, Litz and Kozyrakis, ReFlex: Remote Flash ≈ Local Flash, ASPLOS 2017, sections 4.1 and 5.1–5.3; Table 2

Accessed 17 October 2026. Specification editions are pinned; online implementation documentation can change.