The GPU memory wall: why memory, not compute, binds AI infrastructure

The memory wall is the gap between how fast a processor can compute and how fast — and how much — memory can feed it. The term is not new: computer architects Wulf and McKee named it in 1995, pointing out that processor speed was improving faster than DRAM, and that the gap between two diverging exponentials itself grows exponentially. Their conclusion was that memory, not the processor, would eventually bound performance.

AI accelerators have walked straight into that wall. A current data-center GPU can perform an enormous number of floating-point operations per second, but it can only hold what fits in its on-package HBM and can only feed the compute as fast as that memory's bandwidth allows. On real large-model workloads the expensive part — the compute — spends much of its time waiting on memory. The bottleneck has moved from FLOPS to bytes.

Two distinct limits make up the wall, and they bind differently:

• Capacity (gigabytes of HBM) decides whether the job fits at all.
• Bandwidth (terabytes per second) decides how fast tokens come out once it fits.

What is HBM and why it matters covers both in detail; this page is about what to do when you hit either one.

The clearest way to see the wall is the roofline model: any operation is limited either by compute or by memory, depending on its arithmetic intensity — the number of math operations it does per byte moved from memory. Low intensity means the hardware finishes the math and sits waiting for the next bytes; it is memory-bound.

Large-language-model serving has two phases with opposite character:

• Prefill (processing the prompt) crunches many tokens at once — a matrix-matrix multiply with high arithmetic intensity. It genuinely uses the GPU's compute.
• Decode (generating the answer one token at a time) is a matrix-vector multiply. To produce a single token it must read the model's weights out of HBM — a fixed byte cost no matter how few tokens you process — giving an arithmetic intensity of roughly 1.

Because most production traffic is decode-dominated, the ceiling on tokens per second is set by how fast weights and cache stream out of memory, not by how many FLOPS the GPU can theoretically do. This is why "we bought more compute and throughput barely moved" is such a common story — and why a memory-upgraded part on an older architecture can beat intuition.

Here is the economic distortion the wall creates. Because a model and its working memory must fit in HBM or the job does not run, capacity — not compute — often drives the purchase. Teams add GPUs to add gigabytes, then run decode-heavy traffic that leaves those GPUs' compute units underused. The marginal GPU is a memory purchase wearing a compute price tag.

Two things make this worse:

• HBM cannot be expanded after purchase. It is soldered onto the GPU package. Need more capacity, add another whole GPU.
• HBM is supply-constrained. Memory prices rose sharply through 2025 and high-bandwidth memory has been effectively sold out into 2026, so "just add HBM" is neither cheap nor freely available.

The two ways the wall actually shows up on a cluster are covered in the spokes: KV cache too large for VRAM (the long-context, high-concurrency failure) and stranded DRAM at fleet scale (the fleet-economics failure).

The first responses to the wall are software, and they are real:

• Quantization — store weights and KV cache in fewer bits (FP8, INT4, and KV-specific schemes). Halves or quarters a footprint.
• KV-cache offload and tiering — spill the KV cache out of HBM to CPU DRAM, SSD, or a remote pool.
• Prefill/decode disaggregation and prefix caching — reuse and relocate cached state instead of recomputing it.

Every one of these is worth doing. But note what they are: ways to stretch a fixed HBM budget, not ways to add capacity. Past a point, the offloaded and tiered bytes need somewhere real to land — which is where the hardware response comes in.

The structural fix is to stop treating memory as something permanently welded to one GPU or one server, and instead build a memory pool many servers can draw from. This is memory disaggregation: capacity lives in a shared appliance and is served over a fast network fabric, so a job that needs more memory borrows it from the pool instead of forcing the purchase of another whole GPU or server.

There are two fabrics in play, and it is worth being precise about their status — this is the honest framing, not a strawman: CXL is the low-latency endgame for rack-scale memory semantics; RDMA/RoCEv2 is what you can pool across servers today without waiting for a new fabric generation. For a team hitting the wall now, the Ethernet-based pool is the deployable answer, and the two are complementary rather than rivals.

Pooling memory instead of over-buying GPUs changes the shape of a buildout:

• Capacity decouples from compute. Add memory without adding GPUs; stop paying for FLOPS you only bought to get gigabytes.
• Utilization rises. Memory that would sit stranded on one server becomes available to whichever job needs it — the fleet-economics case in stranded DRAM at fleet scale.
• Long context and high concurrency get somewhere to put the KV cache other than "another GPU" — the case in KV cache too large for VRAM.

The concrete instance of this pattern is a disaggregated-memory appliance: a rack-scale box of high-capacity DDR5 that shares one pool to multiple servers over RDMA/RoCEv2 Ethernet. The TORmem M1000 is exactly that — powered by AMD EPYC and high-capacity DDR5, serving a shared memory pool to many servers over RDMA over RoCEv2 so remote memory behaves close to local, letting AI and in-memory workloads expand memory capacity and utilization without over-provisioning DRAM per server. It rides a low-latency Ethernet fabric — the same class of network built from parts like the TORswitch and pluggable optics.

If you are sizing against a memory ceiling rather than a compute one, tell us the workload, the context and concurrency targets, and the fabric you run — we can help plan a memory-centric configuration and source the appliance to match. Browse the hardware overview for what is available.