Skip to content

Technology card

Shipping

NVM Express

NVMe · NVM Express

How commands, completion queues and PRP/SGL data pointers fit together, from PCIe to fabrics.

Glossary entry on hesela.dev · JSON record · Volatile write cache · SR-IOV functions · Submission queue · Completion queue · Phase Tag · Command Identifier · Do Not Retry · Command Retry Delay · Doorbells · Interrupt coalescing · Completion polling · PRP data pointers · Scatter Gather Lists · Computational storage drive · NVMe compute namespace · IceClave isolation research · Namespace write protection

On this page6 sections

Architecture and evidence

NVMe defines storage commands and transport models for non-volatile memory. Its PCIe path separates submission and completion queues from data buffers; NVMe over Fabrics extends access over network transports. NVMe is a protocol family, not a connector, drive shape or promise of a particular latency.

Volatile write-cache presence, enable state and namespace scope are separate observations. Base 2.3 reports controller presence through VWCP and Get Features 06h with selection 0, 1 or 2 returns the selected WCE value. A namespace with VWCNP set has no volatile write cache despite a controller-level presence indication; section 7.2 explicitly covers the Flexible Data Placement case. Get Features selection 0 requests the current value, while selections 1 and 2 request default and saved values. Selection 3 instead returns supported capabilities: bit 0 means Saveable, not Write Cache Enable. A saved-value request falls back to the default when the feature is not saveable or no saved value exists; a successful read does not prove a saved setting exists. A failed query is not evidence that caching is disabled [10] [12].

A cache whose contents are guaranteed to reach nonvolatile storage on power loss is classified as nonvolatile for the VWC feature. Therefore, absent VWC does not establish a DRAM-less design. Neither a capability flag nor an enabled-state reading is a physical test of power-loss protection [10] [11].

When a present volatile cache is disabled, Base 2.3 requires user data written to a namespace to be persistent. This does not turn several writes into one atomic filesystem transaction. When caching is enabled, Flush covers the specified namespaces' commands completed before Flush submission, not all commands merely submitted earlier. Enabled caching does not imply that each completed write remains volatile [10].

NVMe 1.0 was ratified on 1 March 2011, as recorded in the revision history of NVMe 1.0e. This is a standardization milestone, not a claim about the first commercial SSD. Later specifications distinguish memory-based and message-based transports.

On the PCIe path described by SPDK, the host places commands in a submission ring and updates a doorbell register. Several commands can share one doorbell update. The controller transfers the associated data with DMA and writes completion entries into a separate ring; the host advances its completion head after consuming entries. The command ring is not the ordinary read/write payload buffer.

Queue consumption and command completion are different events. SQHD advances when command slots are consumed, not when all those commands finish. Its value is captured when the completion entry is created and may lag the controller when the host reads it. Each I/O submission queue is associated with one completion queue, but several may share it; the Admin pair remains one-to-one. A completion's SQID and CID identify its command. Submission order does not guarantee completion order, so dependent operations need their applicable command-ordering rules [31].

Command identity has a separate lifetime from ring slots. A CID is scoped to its submission queue, not globally unique. SPDK v24.09 assigns a tracker CID to a request; after non-retry completion processing, the tracker returns to a free list for later reuse. The CID is neither a permanent request identity nor necessarily the submission-ring slot number. An SQHD advance alone does not establish that a command has completed or that its CID can be reused [26] [25].

A full completion queue stops further status posting to that queue. The controller may also pause its associated submission queues until the host returns completion slots; this condition must not stop unrelated submission queues. Host processing therefore matters: reading an entry alone does not release its slot to the controller; the updated CQ head must be communicated [31].

Completion-ring traversal has its own synchronization. Linux v6.12 checks the expected phase bit, uses a DMA read barrier before processing the entry, and flips the expected phase when its head wraps. The phase is not the command's success status. The Phase Tag is reserved in NVMe over Fabrics [26]. With doorbell-buffer support configured, its event check can suppress a PCIe register write, so register-write counts are not command counts [33].

Retry interpretation is separate from completion freshness. Do Not Retry (DNR)=1 indicates that the same command is expected to fail if resubmitted to any controller in the NVM subsystem. DNR=0 only leaves open the possibility of success; it is not an instruction to retry indefinitely or a verdict on the health of the whole SSD. Read DNR alongside the Status Code Type and Status Code. When set, the More flag indicates additional command status in the Error Information log [22].

Command Retry Delay (CRD) is a selector, not a duration. With DNR=0 and Host Behavior Support ACRE=1, CRD=00b permits immediate retry; 01b, 10b and 11b select the controller's CRDT1, CRDT2 and CRDT3 values in 100 ms units. Otherwise CRD is reserved. Waiting is recommended, but an early retry is not a protocol error. The interval is not a completion deadline or a promise of recovery [22].

Linux v6.12 illustrates the timing step: nvme_retry_req selects the advertised delay, converts it to milliseconds and requeues the request. This pinned function is evidence about an implementation after its retry decision, not a universal retry policy or a Hesela hardware experiment [23].

The retry decision can stop before that timing step. In Linux v6.12, nvme_decide_disposition ends a failed request when the block-layer no-retry condition, DNR or the retry limit applies. Later branches distinguish authentication and multipath failover from requeueing. DNR=0 therefore does not establish that this host will retry; a retry delay is meaningful only after the request is eligible [21].

Posting a completion entry and notifying the host are separate steps. With interrupts enabled, a notification prompts the host to inspect the relevant completion queues; it is not a one-command completion record. Interrupt coalescing can let one interrupt cover several completions, reducing notification frequency at the cost of possible delay. Host software still processes the entries and returns their slots [27] [26].

I/O completion polling checks for finished requests without waiting for an interrupt to prompt each check. SPDK's completion-processing call is non-blocking: it can return zero ready completions, allowing other work between calls. SPDK also supports interrupt-assisted operation for documented transports and configurations; wakeup still requires completion processing. Polling therefore does not inherently mean continuous busy-waiting, and SPDK does not universally disable interrupts [28] [29].

Fewer interrupts are not automatically a better application outcome. Tai et al. (OSDI 2021) show a mixed workload where adaptive coalescing raises aggregate throughput while worsening synchronous-read latency. Their calibrated-interrupt proposal conveys request urgency and batch boundaries to the device. The evaluation uses real SSD I/O but emulates interrupt generation with dedicated CPU cores and interprocessor interrupts, rather than modified SSD firmware. This is a research design, not an ordinary NVMe setting or a Hesela reproduction [24].

The paper also exposes an ordering trap: its evaluated Barrier annotation is relaxed, equivalent to Urgent, and does not wait for every earlier submission to finish. Section 4.3 discusses a stricter alternative with extra accounting. This notification annotation must not be read as a completed-write ordering or durability guarantee [24].

Physical Region Page (PRP) entries describe transfer memory, not NAND pages. For ordinary PRP-based Read and Write commands, PRP1 may begin partway through the first page. When PRP1 alone is insufficient, PRP2 identifies the next data page when that is sufficient; otherwise it identifies a list of additional page addresses. The first-page offset therefore changes when a list is needed. NVMe 1.0e defines these cases, and Linux v6.12 implements them in nvme_pci_setup_prps.

List storage has its own alignment rules. NVMe Base 2.0a section 4.1.1 permits an 8-byte-aligned initial list pointer within a memory page; later list pages start on a page boundary. Data-page entries inside lists are page-aligned. With a page-aligned 4 KiB list and 8-byte entries, 512 slots are available, but a non-final list uses one slot for the next-list pointer, leaving 511 data addresses. A nonzero initial-list offset leaves fewer slots. These are list-storage counts, separate from the data-page spans below.

Scatter Gather Lists (SGLs) use typed descriptors, including address-and-length data blocks and segments containing further descriptors. Capability and transport rules constrain their use. In Linux v6.12, the PCIe driver maps a request before constructing either PRPs or SGL descriptors. A Linux scatterlist in that mapping path does not prove the command uses an NVMe SGL. This implementation detail is not a claim that SGLs always improve performance.

For PCIe buffer pointers, device-visible DMA addresses are the relevant addresses. The Linux DMA guide distinguishes them from CPU virtual and CPU physical addresses: an IOMMU can translate the address used by the device. Contiguity in an application's virtual buffer therefore does not establish one contiguous DMA segment.

NVMe over Fabrics uses command and response capsules in a message-based model. Do not transfer the PCIe ring, register-doorbell or data-placement description unchanged to every fabric. Base specification 2.0a sections 2 and 4.1 establish the transport and descriptor distinctions; the cited versions are reference points, not claims about the latest revision.

Separate queues do not guarantee workload isolation. Tavakkol et al. (MQSim, FAST 2018, section 3.1) measured interference between two read flows on four 2016-2017 SSDs. Section 6.1 then uses simulation to separate contention in the write cache, mapping cache and back-end resources. The latter experiments are not direct measurements of every commercial controller. Compare each workload alone and together; report its latency and throughput, not just aggregate IOPS. This is a literature-based testing recommendation, not a Hesela hardware result.

Namespace write protection changes command permissions, not the transport. Under Base 2.3, it is enforced by every controller attached to the protected namespace. Read and reservation operations remain permitted, while Format NVM or Sanitize must fail if their effects would modify protected namespace media. A host must consider the command's full scope, not merely its selected namespace [13]. Entry permission is not the active protection state: the RPMB controls that allow entering until-power-cycle or permanent protection clear on a Controller Level Reset, but that reset does not clear an active permanent protection state [13].

Computational storage is not a capability implied by the NVMe label. SNIA distinguishes a drive containing compute engines and persistent storage from a processor that computes for associated storage without providing persistent data storage. Its architectural category is broader than either device type [19].

For an I/O controller supporting NVMe Computational Programs 1.3, Execute Program is mandatory but Load Program is optional. A compute namespace may offer device-defined programs that the host cannot change, or accept supported downloadable types. Check its Program List and Downloadable Program Types List before assuming a host binary will run there; a shared command interface is not a universal execution environment [20].

Loading a program, activating it and executing it are separate steps. The program must be activated on the target compute namespace before execution. A Program List response describes activation when the query was processed, not a guarantee that the program remains active. Loading does not necessarily reserve compute resources; activation may do so. Host-loaded programs and activation state are cleared on power-on resets. Flush has no effect on compute namespaces, so it cannot preserve this program state across that reset. This is a program-lifecycle constraint, not a statement that every computed output is volatile. Plan recovery for the program state separately from the storage semantics of its output [20].

A command interface does not locate the computation. Flash-Cosmos, a MICRO 2022 research design, performs bitwise logic through NAND-array sensing, unlike an SSD-controller accelerator. It requires specialized chip commands, operand preparation and firmware support; NVMe program support alone does not imply this implementation [6] [5] [7].

Keep its evidence levels separate: physical-chip characterization supports feasibility, while application speedups use a simulation-based system evaluation. Zero observed bit errors do not establish a general SSD reliability guarantee; the paper explicitly qualifies that boundary in footnote 11 [5].

ParaBit exposes why physical placement matters. The MICRO 2021 design computes through NAND latches; its basic form pairs bits in the same MLC cell. Even its location-free variant keeps bitline alignment and, for the described AND/XOR paths, an LSB second operand. Its custom NVMe command encoding is a research interface, not a standard capability [3] [4].

Error handling still needs its own contract. ParaBit applies ECC during operand reallocation, but that does not correct new sensing errors in the subsequent logic. It also disables operand scrambling and encryption. Flash-Cosmos later addresses raw-error exposure and serial sensing; its zero-error observation remains bounded by the tested population. ParaBit's performance study combines simulation and measured result transfer, with a separate chip-error experiment [3] [5].

ECC protects a representation, not an arbitrary computation. A bitwise operation on encoded inputs need not preserve their parity relationships. In a specified systematic Hamming(7,4) construction, fault-free AND followed by decoding returns 96 wrong payloads out of 256 ordered input pairs, although all 256 raw systematic payloads are correct. With the specified extended Hamming(8,4) decoder, 168 pairs are rejected and the other 88 return correctly. These are exact synthetic counts, not NAND failure rates [1].

A hand-checkable witness uses payloads 1 and 8, encoded as 0000111 and 1001011; code position 1 is the rightmost bit. Their AND is 0000011, whose systematic payload is correctly zero. Its syndrome is 1 XOR 2 = 3, so the seven-bit decoder flips position 3 and returns payload 1 instead. No read fault occurred: the operation invalidated parity before the decoder ran. The replay specifies the bit layout, decoder policies and every denominator [1].

XOR preserves these linear encodings, but its input error masks combine. One individually correctable bit error on each operand cancels when positions match and becomes a double error when they differ. Correcting each operand first succeeds in all 111,360 enumerated events across the specified codes, operators and fault regimes. Ideal logic, fixed layouts and at most one input error per operand are assumptions, not a hardware reliability guarantee. The code, independent-algorithm tests and full result table are available in the offline replay; this illustrates an ECC composition boundary, not an NVMe capability or a replication of either flash prototype [1].

Program support and tenant isolation are separate questions. IceClave is a research example: it protects the shared FTL and uses program identities to check flash access. Its read-only mapping region is not unrestricted access to other tenants' data [9].

The author follow-up describes different encryption-counter treatment for read-only and writable memory pages, with integrity verification for SSD DRAM. Its threat model assumes secure program delivery and trusted hardware vendors, and excludes software side-channel attacks. These assumptions matter when interpreting its security claims [9].

Our reading: an NVMe program command interface should not be treated as evidence that a device implements IceClave or equivalent isolation. Check the implementation and its threat model separately. Hesela has not run this prototype or reproduced its performance results [8] [20].

An offload still needs to identify its input. In Adams, Keys and Mesnier's HotStorage 2019 prototype, a virtual object describes extents and length for one operation; the host library handles input synchronization and output-cache details. Their vendor-specific EXEC is a research interface, not NVMe Execute Program [17].

This is not a recipe for bypassing a live filesystem. Linux 6.12's FIEMAP documentation makes block-device access through returned mappings undefined while the filesystem is mounted, warns that mappings can change between queries, and restricts encoded extents. FIEMAP_FLAG_SYNC does not waive those rules. Treat filesystem coordination as a separate integration requirement [18].

SR-IOV separates PCIe function addresses, not physical drives. Linux v6.12 manages VF enablement and driver probing separately; neither step alone assigns a function to a guest. Configure sriov_drivers_autoprobe before enabling VFs: changing it later does not affect functions already probed by a host driver. Check platform DMA isolation and IOMMU-group ownership as well as device capability [14] [16].

For NVMe implementations supporting both SR-IOV and Virtualization Enhancements, PF and VF controllers are primary and secondary controllers. Namespaces remain separate storage resources and may be shared. Flexible queue/interrupt assignment requires an Offline secondary; disabling its primary or applying a Controller Level Reset to that primary takes the secondary Offline again. In Base 2.0e, clearing SR-IOV Control.VF MSE hides the VF controller properties without resetting their values; register visibility and controller reset are distinct. A visible VF therefore does not prove ready I/O, a private namespace or reset independence [15].

PRP page-span examples

The table isolates PRP address-span accounting: pages touched = ceiling((starting offset + transfer bytes) / 4,096). It assumes ordinary PRP-based data transfer, an adequately mapped buffer and a 4 KiB controller memory page. It does not count list storage, NAND operations or elapsed time.

Illustrative PRP layouts with a 4 KiB controller memory page; arithmetic, not a benchmark.
Transfer bytesFirst-page offset (bytes)Data pages spannedPRP2 role
4,09601Not needed for data
4,0965122Second data page
8,19202Second data page
8,1925123PRP list

Uncertainty notes

  • The date refers to NVMe 1.0 ratification, verified in the NVMe 1.0e revision history.
  • PRP/SGL examples describe transfer-buffer layout, not flash geometry, end-to-end zero-copy or measured throughput. Descriptor layout also does not establish that a controller accepts a given transfer size.
  • The 4 KiB page is an explicit example assumption; controller memory-page size is not universally fixed to it.
  • PRP/SGL sources checked 17 October 2026; queue lifecycle sources checked 20 October 2026 and completion identity sources checked 23 October 2026, Europe/Warsaw. No device benchmark or power-loss test was performed.

Sources

  1. 01Hesela: ECC operation order, exact Hamming-code construction, method and results. · accessed 2026-11-18
  2. 02Hesela: offline ECC operation-order replay, code, tests and results (ZIP). · accessed 2026-11-18
  3. 03Gao et al. ParaBit: Processing Parallel Bitwise Operations in NAND Flash Memory based SSDs. MICRO 2021, pp. 59-70, sections 4.2-4.4, 5.1 and 5.8. DOI: 10.1145/3466752.3480078. · accessed 2026-11-18
  4. 04Hesela corpus: ParaBit, operand placement and error handling. · accessed 2026-11-18
  5. 05Park et al. Flash-Cosmos. MICRO 2022, pp. 937-955, sections 5-7 and footnote 11; DOI 10.1109/MICRO56248.2022.00069. · accessed 2026-11-18
  6. 06Park et al. Flash-Cosmos: MICRO 2022 author presentation, in-flash versus in-storage processing. · accessed 2026-11-18
  7. 07Hesela corpus: Flash-Cosmos, mechanism and evidence boundaries. · accessed 2026-11-18
  8. 08Kang et al. IceClave: A Trusted Execution Environment for In-Storage Computing. MICRO 2021, pp. 199-211, sections 3-6; DOI 10.1145/3466752.3480109. · accessed 2026-11-14
  9. 09Xue et al. Building A Trusted Execution Environment for In-Storage Computing. Author follow-up, arXiv:2205.06361v1, sections 2-3 (2022). · accessed 2026-11-14
  10. 10NVM Express Base 2.3, Figures 196, 199, 328, 335, 408; sections 4.4, 5.2.11.1, 5.2.26.1.4 and 7.2. Copyright 2025 NVM Express, Inc., all rights reserved. · accessed 2026-11-11
  11. 11Microsoft: NVME_CDW11_FEATURE_VOLATILE_WRITE_CACHE, enable state and protected-cache scope. · accessed 2026-11-11
  12. 12nvme-cli v2.11: nvme-get-feature, current/default/saved selection. · accessed 2026-11-11
  13. 13NVM Express Base 2.3: sections 5.2.26.1.35 and 8.1.17, Figures 671 and 690; copyright 2025 NVM Express, Inc., all rights reserved · accessed 2026-11-08
  14. 14Linux v6.12: PCI Express I/O Virtualization Howto, sections 3.1.1, 3.2.1 and 3.3.1 · accessed 2026-10-31
  15. 15NVM Express Base 2.0e (29 July 2024), sections 2.4.1 and 8.26; copyright 2008-2024 NVM Express, Inc., all rights reserved · accessed 2026-11-02
  16. 16Linux v6.12: VFIO, Groups, Devices, and IOMMUs; Device cdev ownership · accessed 2026-10-31
  17. 17Adams, Keys and Mesnier, Respecting the block interface - computational storage using virtual objects, HotStorage 2019, sections 3.1-3.2. · accessed 2026-11-01
  18. 18Linux 6.12, Fiemap Ioctl: Request Basics and Extent Mapping, including FIEMAP_EXTENT_ENCODED. · accessed 2026-11-01
  19. 19SNIA Computational Storage Architecture and Programming Model 1.2, definitions 3.1.3-3.1.10; copyright 2026 SNIA · accessed 2026-10-31
  20. 20NVM Express Computational Programs Command Set 1.3 (31 July 2026); copyright 2008-2026 NVM Express, Inc., all rights reserved. · accessed 2026-11-01
  21. 21Linux v6.12, nvme_decide_disposition: stop, authenticate, fail over or retry · accessed 2026-10-23
  22. 22NVM Express Base 2.3, section 4.2.3, Figure 100 and Identify Controller Figure 328: DNR, CRD and CRDT1-3 · accessed 2026-10-23
  23. 23Linux v6.12, nvme_retry_req: CRD selection, 100 ms conversion and requeue · accessed 2026-10-23
  24. 24Tai, Smolyar, Wei and Tsafrir, Optimizing Storage Performance with Calibrated Interrupts, OSDI 2021, sections 3.3, 4.2.1, 4.3 and 5.1 · accessed 2026-10-23
  25. 25SPDK v24.09: PCIe tracker retirement and CID assignment · accessed 2026-10-23
  26. 26NVM Express Base 2.0d, sections 3.3.1.2, 3.3.3 and 5.27.1.6-7; copyright 2008-2024 NVM Express, Inc., all rights reserved · accessed 2026-10-23
  27. 27NVM Express PCIe Transport 1.0d, sections 3.2, 3.4.1 and 3.5; copyright 2008-2023 NVM Express, Inc., all rights reserved · accessed 2026-10-20
  28. 28SPDK: spdk_nvme_qpair_process_completions API contract · accessed 2026-10-20
  29. 29SPDK: NVMe Interrupt Mode, supported transports and operational notes · accessed 2026-10-20
  30. 30NVM Express 1.0e: revision history and PRP-based Read/Write; copyright 2007-2014 NVM Express, Inc., all rights reserved · accessed 2026-10-17
  31. 31NVM Express Base 2.0a, sections 2, 3.1.1, 3.3.1.2, 3.3.3.2 and 4.1; copyright 2007-2021 NVM Express, Inc., all rights reserved · accessed 2026-10-20
  32. 32SPDK: Submitting I/O to an NVMe Device · accessed 2026-10-17
  33. 33Linux v6.12: NVMe PCIe data mapping, queue handling and doorbell-buffer checks · accessed 2026-10-20
  34. 34Tavakkol et al., MQSim, FAST 2018: sections 3.1, 6.1 and Appendix A · accessed 2026-10-17
  35. 35Linux 6.12: Dynamic DMA mapping guide, CPU and DMA addresses · accessed 2026-10-17

Published by Hesela team

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. NVM Express. Hesela. https://hesela.com/tech/nvme/