KV cache too large for VRAM: why long-context inference runs out of memory
TL;DR
The KV cache is the per-request attention state a transformer keeps so it does not recompute the whole prompt for every new token. It grows linearly with context length and linearly with the number of concurrent requests, and it lives in the same HBM as the model weights. At short context it is a rounding error; at long context and real concurrency it becomes the dominant consumer and is usually what triggers the out-of-memory error. Quantization and offload stretch the budget, but past a point the cache needs somewhere real to land — CPU DRAM, or a disaggregated memory pool the whole cluster can share.
On this page
What the KV cache is and why it exists
When a transformer generates text, every new token attends to every previous token. Without a cache the model would recompute the key and value vectors for the entire sequence at every step — quadratic, ruinous work. The KV cache stores the key (K) and value (V) vectors for tokens already processed, so each new token only computes its own K and V and reuses the rest. It is what makes autoregressive decoding affordable.
The catch: that cache lives in the GPU's HBM, right next to the model weights, and it is per request and grows with every token generated.
The size formula, and why it explodes
KV cache size is exact and easy to compute:
bytes = 2 × n_layers × n_kv_heads × d_head × bytes_per_element × batch_size × sequence_length
- The leading 2 is one K and one V per position.
- n_layers, n_kv_heads, d_head are fixed by the model architecture (grouped-query attention cuts n_kv_heads, which is why modern models use it).
- bytes_per_element is the precision: 2 for FP16/BF16, 1 for FP8.
- The two terms that move at serving time are sequence_length (context) and batch_size (concurrency) — and the cache is linear in both.
Linear-in-both is the whole problem. Double the context, double the cache. Double the number of simultaneous users, double it again. A 70B-class model at 16-bit costs on the order of a third of a megabyte per token — tens of gigabytes of KV cache for a single very-long-context sequence, before you multiply by concurrency. Even a small model at a long context and modest batch can produce a KV cache larger than fits alongside the weights on many single GPUs.
Why it is the KV cache, not the weights, that OOMs
Model weights are a fixed cost: load them once, they do not grow. The KV cache is a variable cost that scales with exactly the two things product teams keep pushing up — longer context windows and more concurrent users. So the HBM budget looks like:
HBM = weights (fixed) + KV cache (grows with context × concurrency) + activations + overhead
At short context and batch 1, the KV term is negligible and weights dominate. At long context and real concurrency, the KV term overtakes the weights and is what blows the budget. This is the mechanism behind "it worked in the demo and OOM'd in production" — the demo was short-context, single-user; production is long-context, many-user.
It is also why the GPU memory wall bites hardest on modern, long-context products: the industry raised context windows to hundreds of thousands of tokens and the KV cache grew linearly to match.
The software levers (do these first — they stretch, they do not add)
Before buying anything, exhaust the software budget:
- KV-cache quantization — store K and V in fewer bits, with schemes designed to hold quality.
- Eviction / sparsity — keep only the tokens that matter. Lossy, but effective on many workloads.
- Grouped-query / multi-query attention — fewer KV heads, smaller cache; a model-architecture choice you inherit rather than tune.
- Paged KV and prefix caching — paging removes fragmentation waste and lets common prefixes be reused across requests instead of re-cached.
These are all real and all worth doing. But every one of them rescales a fixed HBM budget; none of them adds a gigabyte. When context and concurrency keep climbing, you run out of levers.
Where the overflow can actually go
When the cache no longer fits in HBM even after quantization, the next question is where does it spill? — because the alternative, adding another whole GPU just to hold cache, is the expensive habit the memory wall creates.
The production answer is KV-cache offload to a memory tier outside the GPU. The first tier is CPU DRAM on the same host. Beyond that, KV-cache-centric serving stacks move cache blocks across the cluster over RDMA, pooling memory that lives outside any single GPU. The tiering hierarchy that emerges is local HBM → local DDR → pooled DDR (over RDMA/RoCEv2) → SSD. The disaggregated pool is the new middle tier — far larger than any GPU's HBM, far faster than storage, and shared across servers so one job's spare capacity serves another's overflow.
That pool is a hardware product: a disaggregated-memory appliance — a rack of high-capacity DDR5 shared over RDMA/RoCEv2 Ethernet — is where offloaded KV cache can land at scale without each server over-provisioning HBM. The TORmem M1000 is a disaggregated-memory appliance of this kind — powered by AMD EPYC and high-capacity DDR5, serving a shared memory pool to many servers over RDMA over RoCEv2 so remote memory behaves close to local. It rides a low-latency Ethernet fabric of the kind built from the TORswitch and pluggable optics.
If long-context serving is forcing you to buy GPUs just to hold KV cache, tell us the model, the context and concurrency targets, and the fabric you run — we can help plan a memory-pool configuration and source the appliance. See the GPU memory wall for the full picture, or the hardware overview.
Frequently asked questions
Why does my KV cache get so big at long context?
Because it grows linearly with sequence length — every token you add stores one more K and one more V vector in every layer. A 70B-class model at 16-bit costs roughly a third of a megabyte per token, so a very long context alone is tens of gigabytes for a single request, before you multiply by concurrency.
Is it the weights or the KV cache that runs me out of memory?
At long context and real concurrency it is usually the KV cache. Weights are a fixed one-time cost; the KV cache scales with the two things you keep increasing — context length and simultaneous requests — so it eventually overtakes the weights and triggers the OOM.
How do I reduce KV cache memory without buying GPUs?
Quantize K and V to fewer bits, use grouped/multi-query attention, evict low-importance tokens, and page the cache to remove fragmentation. These stretch the budget. When they run out, offload the cache to CPU DRAM or a disaggregated memory pool over RDMA rather than adding GPUs.
Can KV cache live outside GPU memory?
Yes. Production serving stacks move KV blocks to CPU DRAM and to remote memory pools over RDMA. Remote memory is higher latency than HBM but far larger and shareable across the cluster, which is the point of a disaggregated memory pool.
Related
What is the GPU memory wall?
The "memory wall" is a thirty-year-old observation — compute speed improves faster than memory speed and capacity, so eventually memory, not the processor, sets the ceiling. In AI infrastructure that ceiling has arrived: a modern GPU can do far more math per second than its fixed on-package HBM can hold or feed, so large-model inference is usually memory-bound, not compute-bound. The practical symptom is teams buying more GPUs than their compute needs just to get memory capacity, then watching that compute sit idle. The structural response is to stop pinning every byte to one GPU or server and instead pool memory — a disaggregated pool shared across servers over a fast fabric.
Read →What is stranded DRAM, and why is half my memory idle?
Every server is bought with enough DRAM for its worst-case moment, so on average a large fraction of that memory is idle — "stranded" on a CPU that is not using it while a neighbor a rack away is memory-starved. Hyperscale measurements put the waste around a quarter to a half of all DRAM. Because memory is one of the biggest and now fastest-rising lines in a server's cost, stranded DRAM is real capital sitting dark. The fix is the same as for the GPU memory wall: stop welding memory to individual servers and pool it, so idle capacity in one place serves demand in another.
Read →What is HBM and Why Does It Matter for AI?
HBM is high-bandwidth memory — DRAM stacked vertically and mounted next to the GPU die on the same package, giving far more bandwidth than the GDDR used on graphics cards. It matters because large-model inference is usually memory-bound, not compute-bound: capacity decides whether a model and its KV cache fit at all, and bandwidth decides how fast tokens come out. For most buyers, "how many GPUs do I need" is really "how much HBM do I need, and how fast".
Read →How to Size a GPU Cluster
Size a cluster in one direction: workload first, then memory footprint, then GPU count, then node count, then the fabric, and only then power, cooling and floor space. The step most people miss is the last one — nameplate GPU TDP is not facility load. Host platforms, networking, storage and cooling overhead typically push the number at the meter to roughly two-and-a-half to three times the sum of the GPU TDPs. That is why AI buildouts turn into power procurement projects.
Read →Last updated