NVMe multipath: ANA access states and Linux path policies

One group, different controller views

An ANA group collects namespaces that undergo access-state transitions together relative to a controller. Its identifier is scoped to the NVM subsystem. The same group can be optimized through controller A and non-optimized through B. A group is neither a physical cable nor an independent failure domain. [1, section 8.1.1]

Access preference comes before load balancing

Linux native multipath combines paths to the same namespace within an NVM subsystem into one block device. NUMA favors locality, round-robin rotates eligible paths, and queue-depth considers active I/O. All prefer optimized over non-optimized access. Round-robin therefore does not promise equal traffic across every connected controller. [3]

In Linux v6.12, queue-depth uses controller-level nr_active counters, not just this namespace's outstanding requests or an individual hardware queue's occupancy. It is not a measured-latency ranking. An idle non-optimized route does not outrank a busy optimized route. [2, nvme_queue_depth_path]

The module iopolicy supplies a default copied into a subsystem. For an existing subsystem, inspect /sys/class/nvme-subsystem/nvme-subsysX/iopolicy; reading only the module default does not establish the policy actually in use. This distinction is pinned to v6.12, not every distribution's backports. [2]

Optimized access is not a latency measurement

Gunawi et al. studied 101 operator-reported fail-slow incidents from 12 institutions, covering 2000-2017. A report could represent repeated incidents; the collection was not a random sample or a device-time denominator. It supports the existence of performance faults, not an NVMe ANA failure rate. [6, section 2]

The study describes how retries and error correction can preserve service while making it slow. It also argues for monitoring across hardware layers. It did not evaluate ANA or Linux v6.12 path policies. [6, sections 3.2 and 6.2]

Hesela's inference: keep access state and observed performance as separate evidence. An optimized label cannot establish that a path has the lowest latency under the current workload. Compare like-for-like I/O sizes, read/write mixes and observation windows before attributing a slowdown to path selection. See latency measurement limits.

Collect topology and access evidence separately

nvme list-subsys reports subsystem and controller topology; a device argument limits the report to controllers and subsystems leading to that device. Keep the selected namespace, controller identifiers and collection time with the output. A topology snapshot is not a failover measurement. [5]

nvme ana-log retrieves the ANA log from a controller character device. The --groups option omits the namespace listing, so that output alone cannot reconstruct group membership. Compare reports from the relevant controllers rather than treating one response as a universal view. [4]

Capacity reports also need controller context. Base 2.3 specifies that some Identify Namespace fields, including NUSE and NVMCAP, are cleared when accurate capacity information is unavailable in inaccessible or persistent-loss states. Use an optimized or non-optimized controller's report; these zeros do not establish that the namespace is empty. [1, sections 8.1.1.6-8.1.1.7]

These are inspection interfaces, not instructions to disconnect production paths. Hesela's review checklist is to record the kernel and tool versions, namespace identity, group membership, each controller's state and the effective subsystem policy before interpreting an incident.

What this does not guarantee

If no eligible path exists, Linux can requeue I/O or fail it. A usable alternate path is not a guarantee of interruption-free recovery. [2, nvme_ns_head_submit_bio]

Architectural implication: alternative routes are not additional copies of the data. Check shared switches, controllers and backing media before claiming independent failure protection. ANA also does not fence a stale writer or settle whether a timed-out write completed. See storage fencing and uncertain completion after timeouts.

We reviewed pinned documents and code; we did not perform hardware experiments. No latency bound, availability percentage or durability claim follows from this review.

Corpus: ANA group, NVMe multipath I/O policy, ANA, NVMe multipath and NVMe namespace.

Primary sources

1. NVM Express Base Specification 2.3 (1 August 2025), section 8.1.1: Asymmetric Namespace Access. Pinned edition, not a claim about the latest revision.
2. Linux v6.12, drivers/nvme/host/multipath.c: ANA selection, controller nr_active accounting and subsystem iopolicy.
3. Linux kernel documentation: native NVMe multipath policies.
4. nvme-cli v2.11: nvme-ana-log(1), controller log request and groups-only option.
5. nvme-cli v2.11: nvme-list-subsys(1), subsystem/controller topology and device filtering.
6. Gunawi et al. (FAST 2018), Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems, pp. 1-14; methodology and monitoring limits.

Base 2.3 is the reviewed edition, not a claim about the latest standard. No publisher text, figure or hardware trace is reproduced.