Stranded DRAM at fleet scale: why half your server memory sits idle
TL;DR
Every server is bought with enough DRAM for its worst-case moment, so on average a large fraction of that memory is idle — "stranded" on a CPU that is not using it while a neighbor a rack away is memory-starved. Hyperscale measurements put the waste around a quarter to a half of all DRAM. Because memory is one of the biggest and now fastest-rising lines in a server's cost, stranded DRAM is real capital sitting dark. The fix is the same as for the GPU memory wall: stop welding memory to individual servers and pool it, so idle capacity in one place serves demand in another.
On this page
What "stranded" means
Memory is stranded when it is installed in a server, powered, and paid for, but cannot be used by any workload that needs it — because it is physically bolted to a CPU (or a VM) that happens not to need it right now. It is not broken and it is not free; it is simply unreachable by the job one rack over that is short on memory. Multiply that across thousands of servers and a large fraction of a fleet's most expensive component is doing nothing.
Why over-provisioning is structural, not sloppy
Stranding is not a procurement mistake; it falls out of how servers are sized. You buy each machine with enough DRAM for its peak — the worst-case working set — because a job that runs out of memory fails, and you cannot add DRAM to a running server. But workloads rarely sit at peak. The gap between peak-provisioned and average-used is idle by construction, and it exists on every machine at once.
Two forces widen the gap:
- Fixed ratios. Servers ship with a fixed DRAM-per-core ratio chosen for the hungriest tenant, so memory-light workloads over-provision massively.
- Isolation. In a virtualized fleet, memory rented to one VM cannot be lent to another even when idle — it is stranded inside the rental boundary.
What the hyperscale research actually measured
This is measured, not hand-waved. Microsoft Azure's Pond study (ASPLOS 2023) instrumented production cloud traces and found:
- 50% of VMs never touch 50% of the memory they rent — that memory is stranded inside the rental.
- Up to about 25% of total DRAM is stranded at any given moment.
- DRAM is more than 50% of server cost — so stranded memory is stranded capital, not a rounding error.
- Pooling across just 8–16 CPU sockets captures most of the achievable benefit; you do not need a giant pool.
Industry restatements land in the same band: most enterprise DRAM sits locked to CPUs at roughly 50–60% utilization. The precise number varies by fleet and workload; the direction does not.
Why this got expensive in the AI era
Stranded DRAM was always wasteful, but two things turned it from a footnote into a capital problem:
- Memory prices surged. DRAM and HBM prices rose sharply through 2025 in the AI memory supercycle, with high-bandwidth memory effectively sold out into 2026. Idle memory you already own is worth more, and replacing it costs more.
- AI made memory the dominant line. As the GPU memory wall pushes teams to buy capacity they do not compute against, the share of spend going to memory rises — and so does the absolute cost of leaving it stranded.
Put together: a fleet is paying premium prices for its most expensive component and using a fraction of it. That is the economic case for changing how memory is attached to servers.
Pooling: reclaim the idle instead of buying more
The fix mirrors the GPU-side answer: disaggregate the memory. Instead of each server owning a fixed, over-provisioned DRAM allotment, capacity moves into a shared pool that many servers draw from on demand. Idle memory in one place serves a shortfall in another, so the fleet is sized closer to its aggregate need rather than the sum of every server's individual peak — which is where the stranding lived.
Two fabrics carry the pool, and their status differs (the full comparison is in the hub): CXL memory pooling is the low-latency, load/store approach, real and advancing, but multi-host pooling across servers is still maturing toward fleet-scale deployment. RDMA / RoCEv2 disaggregation is a pool of DDR5 shared to many servers over standard lossless Ethernet, deployable today with the network you already run.
For a fleet that wants to reclaim stranded DRAM now, the Ethernet-based pool is the deployable path. A disaggregated-memory appliance — a rack of high-capacity DDR5 shared over RDMA/RoCEv2 — lets servers expand memory capacity and utilization without over-provisioning DRAM per server, which is precisely the stranding it is meant to end. The TORmem M1000 is that appliance: powered by AMD EPYC and high-capacity DDR5, serving a shared pool to multiple servers over RDMA over RoCEv2 so remote memory behaves close to local. It rides a low-latency Ethernet fabric of the kind built from the TORswitch and pluggable optics.
If your utilization dashboards show memory provisioned but idle across the fleet, tell us the server shapes, the workloads, and the fabric you run — we can help plan a pooled-memory configuration and source the appliance. Start with the GPU memory wall, or browse the hardware overview.
Frequently asked questions
What does "stranded DRAM" mean?
Memory that is installed, powered, and paid for but cannot be used by a workload that needs it, because it is physically tied to a CPU or VM that is not using it. It is idle capacity you already own but cannot reach from where the demand is.
How much server memory is actually idle?
Hyperscale research puts it high: Microsoft's Pond study found half of VMs never touch half their rented memory and up to about 25% of DRAM is stranded at any moment; industry restatements cite 50–60% CPU-locked utilization. The exact figure depends on the fleet and workload, but the direction is consistent.
Why not just buy less memory per server?
Because each server must cover its own worst-case moment or its jobs fail, and you cannot add DRAM to a running machine. Sizing every server for peak, independently, guarantees idle memory in aggregate. Pooling lets you size for the fleet's combined need instead of the sum of individual peaks.
How does memory pooling reclaim stranded DRAM?
It moves capacity into a shared pool that many servers draw from on demand, so idle memory in one place serves a shortfall in another. CXL is the low-latency version still maturing for multi-host use; an RDMA/RoCEv2 disaggregated-memory appliance does it over standard Ethernet today.
Related
What is the GPU memory wall?
The "memory wall" is a thirty-year-old observation — compute speed improves faster than memory speed and capacity, so eventually memory, not the processor, sets the ceiling. In AI infrastructure that ceiling has arrived: a modern GPU can do far more math per second than its fixed on-package HBM can hold or feed, so large-model inference is usually memory-bound, not compute-bound. The practical symptom is teams buying more GPUs than their compute needs just to get memory capacity, then watching that compute sit idle. The structural response is to stop pinning every byte to one GPU or server and instead pool memory — a disaggregated pool shared across servers over a fast fabric.
Read →Why is my KV cache too large for VRAM?
The KV cache is the per-request attention state a transformer keeps so it does not recompute the whole prompt for every new token. It grows linearly with context length and linearly with the number of concurrent requests, and it lives in the same HBM as the model weights. At short context it is a rounding error; at long context and real concurrency it becomes the dominant consumer and is usually what triggers the out-of-memory error. Quantization and offload stretch the budget, but past a point the cache needs somewhere real to land — CPU DRAM, or a disaggregated memory pool the whole cluster can share.
Read →What is HBM and Why Does It Matter for AI?
HBM is high-bandwidth memory — DRAM stacked vertically and mounted next to the GPU die on the same package, giving far more bandwidth than the GDDR used on graphics cards. It matters because large-model inference is usually memory-bound, not compute-bound: capacity decides whether a model and its KV cache fit at all, and bandwidth decides how fast tokens come out. For most buyers, "how many GPUs do I need" is really "how much HBM do I need, and how fast".
Read →How to Size a GPU Cluster
Size a cluster in one direction: workload first, then memory footprint, then GPU count, then node count, then the fabric, and only then power, cooling and floor space. The step most people miss is the last one — nameplate GPU TDP is not facility load. Host platforms, networking, storage and cooling overhead typically push the number at the meter to roughly two-and-a-half to three times the sum of the GPU TDPs. That is why AI buildouts turn into power procurement projects.
Read →Last updated