Technology card
ShippingCXL Memory Expansion
CXL-attached memory
How CXL expands memory, how Type-3 devices are addressed, and what changes when capacity is volatile, persistent, pooled or shared.
Glossary entry on hesela.dev · JSON record · Type-3 device · HDM decoder · Device DAX · Memory pooling · Memory sharing · Persistence domains and DAX
Related
On this page6 sections
In one minute
CXL memory expansion adds device-attached memory to a host's address space for coherent access through the CXL.mem protocol.
CXL memory expansion addresses a memory-capacity problem. Start by asking whether the application needs more working memory or persistent storage. Unbacked DRAM remains volatile when attached through CXL. Devices can instead offer persistent capacity, or both kinds, so check the actual memory and recovery contract. [7]
CXL uses PCIe physical infrastructure, but CXL.mem is a memory-access protocol, not an NVMe block command. Physical fit and link width do not establish interoperability. [12]
A Type-3 memory expander implements CXL.io and CXL.mem without CXL.cache. This does not mean the host cannot cache its contents: the two protocols describe different directions of coherent access. A Type-2 accelerator additionally has the CXL.cache role. [8]
HDM decoders connect addresses to capacity. In the Linux 6.17 decoder model, root and switch decoders route host physical addresses unchanged; endpoint decoders translate them into device physical addresses. The memory region ties together the ranges and interleave configuration. [9]
Linux 6.17 can expose /dev/daxN.Y for an application to mmap() and manage directly, or use dax_kmem to convert capacity into memory blocks for the kernel page allocator. Firmware, build and boot settings can instead expose system RAM during early boot. NUMA association alone does not establish page-allocator availability, and none of these access modes guarantees persistence. The /dev/cxl/memN interface is a management mailbox rather than the application's memory mapping. [10] [11] [9]
Device DAX, filesystem DAX and online System RAM are different access arrangements. Device DAX gives an application a directly mapped region; filesystem DAX supplies file mappings. For the dax_kmem path, hotplug registration and onlining are separate steps: an offline block is not available to the page allocator. A NUMA identifier alone therefore does not prove usable allocator capacity. The daxctl v80 manual also warns that mode reconfiguration destroys existing data; changing access mode is not a recovery procedure. [10] [3] [6] [4]
Memory-map metadata also consumes capacity. In Linux 6.17, when the global and per-device memmap_on_memory settings are both enabled, descriptors occupy part of the hotplugged memory itself. Otherwise the memory map consumes local ZONE_NORMAL capacity. A large movable pool therefore has a metadata-placement choice, with performance consequences if high-latency metadata is frequently accessed. [6]
Adding capacity does not promise that it can later be removed. Linux 6.17 advises treating memory onlined into ZONE_NORMAL as permanently attached to the page allocator. ZONE_MOVABLE aims to retain removability, not a guarantee that every removal will succeed. The DAX hotplug path marks capacity as driver-managed and excludes it from kexec placement to avoid device-reset hazards; this is a kernel-placement rule, not an application data-retention promise. [6]
Read benchmark methods before comparing latency figures. In a 2023 CXL study, MLC used serialized pointer-chasing, whereas memo timed batches of 16 accesses and selected the median of 10,000 repetitions. These results are not interchangeable estimates of a single dependent access. For a memory-tier decision, match the access dependency, workload and placement policy to the application. [5]
A lower observed latency can reflect cache placement, not faster CXL media. On the study's Sapphire Rapids system in four-way sub-NUMA clustering (SNC), a 32 MB buffer exceeded one SNC node's last-level cache but fit the socket-wide cache. MLC reported 41 ns for CXL-A placement versus 76.8 ns for local DDR. CXL-backed lines could use more cache slices; those figures do not isolate off-chip memory latency. Other active SNC nodes could erode that cache advantage. [5]
Pooling and sharing are distinct. CXL 2.0 pooling can allocate different portions of a pool to different hosts, but each physical location can be mapped to at most one host at a time. CXL 3.0 adds cross-host sharing, whose coherence may be managed by software or hardware. Sharing does not automatically imply hardware coherence and does not replace application synchronization or crash-consistency protocols. Verify the revision and implemented features across the complete system; neither feature follows merely from a CXL label. [13] [14] [1]
For shared fabric-attached memory, the fabric manager selects the coherence model. Hardware coherence requires device tracking, an HDM-DB mapping and the back-invalidation channel. Software coordination of cache-line ownership is outside the specification. [1]
Version labels need context: the Consortium documents an XConn prototype combining a CXL 2.0 switch with software-supported sharing. That vendor demonstration is not proof of standardized CXL 3.0 hardware coherence on all CXL 2.0 systems. [2]
Micron's qualification-milestone announcement states that CZ120 modules are available in production volume through OEM channels. That supports the shipping classification for memory expansion, not a claim that every fabric feature is widely available. Its product performance claims are not measurements of an arbitrary CXL configuration. [15]
Compared to neighbors
Choose the host access model first, then establish persistence and performance separately.
| Question | Evidence to check |
|---|---|
| What backs the capacity? | Volatile or persistent capacity reported for the actual device. [7] |
| How does software use it? | Application-managed Device DAX, dax_kmem conversion, or early-boot system RAM in the chosen Linux configuration. [10] [11] |
| Which host can access it? | Exclusive pool allocation versus concurrent shared-region access; check the selected coherence model. [13] [14] [1] |
Uncertainty notes
- Source-backed architecture guide, not a benchmark or tested deployment recipe.
- Linux behavior is scoped to version 6.17 documentation; it is not an October 2026 compatibility matrix.
- Coherency, byte addressability and persistence answer different questions; none alone establishes a recoverable application transaction.
Sources
- 01CXL Consortium: Flash Memory Summit tutorial (August 2023), slides 27-28; pooled/shared FAM and two coherence models. · accessed 2026-11-16
- 02CXL Consortium: XConn CXL 2.0 memory pooling/sharing prototype (30 November 2023); software-supported demonstration. · accessed 2026-11-16
- 03Linux 6.17: Direct Access for files, page-cache bypass and file mappings. · accessed 2026-11-11
- 04daxctl v80: reconfigure-device, destructive conversion and separate onlining. · accessed 2026-11-11
- 05Sun et al.: Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices, arXiv v4 (2023), sections 4.1, 4.3, 5.3 and 6.2. · accessed 2026-10-29
- 06Linux 6.17: Memory Hotplug, memory-map placement. · accessed 2026-10-28
- 07Linux kernel: Devices and Protocols, CXL memory and Type-3 devices. · accessed 2026-10-27
- 08CXL Consortium: Introducing CXL 3.1, webinar Q&A (2024), traditional device types. · accessed 2026-10-27
- 09Linux kernel: CXL Driver Operation, Decoders and Regions. · accessed 2026-10-27
- 10Linux kernel: DAX Driver Operation, device DAX and dax_kmem. · accessed 2026-10-27
- 11Linux 6.17: Linux Init, firmware, build and boot settings for system RAM. · accessed 2026-10-27
- 12CXL Consortium: Supporting Persistent Memory, webinar Q&A, physical interface and latency. · accessed 2026-10-27
- 13CXL Consortium: CXL 2.0 memory pooling, webinar Q&A, part 2. · accessed 2026-10-27
- 14CXL Consortium: Compute Express Link 3.0 white paper, pages 3-4, sharing and fabrics. · accessed 2026-10-27
- 15Micron: CZ120 qualification milestone and production-volume availability through OEM channels. · accessed 2026-10-27