What is HBM and Why Does It Matter for AI?

HBM stands for high-bandwidth memory. It is DRAM built as a vertical stack of dies and mounted on the same package as the GPU, connected by an extremely wide interface. That physical arrangement — short distances, thousands of signal lines — is what lets a modern accelerator move multiple terabytes per second between memory and compute.

Every data-center training and large-model inference GPU uses it. NVIDIA’s Hopper and Blackwell parts use HBM3 and HBM3e; AMD’s Instinct MI300-series does too. Consumer and mid-range inference cards (the L40S, the RTX PRO 6000 class) use GDDR instead, which is cheaper and slower.

The reason a buyer needs to care is simple: on most real workloads, the GPU is waiting on memory, not on math. Two numbers describe the memory, and they constrain you in different ways.

Capacity (in GB) decides whether the job runs at all. Model weights, the KV cache for every in-flight request, activations, and framework overhead all live in HBM. If they do not fit, you either shard the model across more GPUs, quantize it to a smaller number format, or cut the context length. There is no third option — HBM is not expandable after purchase.

Bandwidth (in TB/s) decides how fast it runs. Autoregressive decoding reads a large fraction of the model weights out of memory for every single token generated. That makes token generation a memory-bandwidth problem far more often than a FLOPS problem — which is why a GPU with the same compute but faster memory can serve meaningfully more tokens per second.

The practical consequence: a part with more capacity but similar bandwidth (H200 over H100) lets you serve bigger models and longer contexts. A part with more bandwidth lets you serve the same model faster. Most buyers need both, in that order.

Published per-GPU memory capacity and peak bandwidth across the accelerators most buyers cross-shop. Note the two jumps that matter: H100 → H200 is a memory-only upgrade on the same Hopper compute, and Blackwell → Blackwell Ultra is another large capacity step at similar bandwidth.

Serving a large language model has two phases with very different characteristics.

• Prefill — processing the prompt. This is a dense matrix operation over many tokens at once, and it genuinely uses the GPU’s compute. Long prompts are compute-heavy.
• Decode — generating the response, one token at a time. Each step reads the weights it needs out of HBM to produce a single token. Arithmetic intensity is low; the GPU spends most of its time streaming memory.

Because most production traffic is dominated by decode, the ceiling on tokens per second is usually set by how fast weights and cache stream out of HBM. That is why "we bought more FLOPS and throughput barely moved" is such a common story, and why a memory-upgraded part on an older architecture can beat intuition. Measure where your inference actually stalls before you buy up a generation.

You can estimate the HBM a workload needs from three inputs: parameter count, number format, and context length.

Weights = parameters × bytes per parameter. A 70-billion-parameter model at FP16/BF16 (2 bytes) needs about 140 GB just for weights. The same model at FP8 (1 byte) needs about 70 GB. Quantization is the cheapest lever you have.

KV cache = the per-request attention state, and it grows linearly with context length and with concurrency. For a 70B-class model in the common shape (80 layers, 8 key/value heads under grouped-query attention, 128-dimension heads) the cache costs roughly 0.33 MB per token at 16-bit precision. At a 128,000-token context that is roughly 40 GB — for one sequence. Multiply by the number of concurrent requests you intend to serve.

Overhead — activations, framework and runtime allocations, fragmentation. Leave headroom; a plan that fits exactly does not fit.

Run that arithmetic and the shape of the answer appears quickly. That 70B model at FP16 does not fit on one 80 GB H100 at all. It fits on a single 141 GB H200 only after quantizing to FP8 — and then the remaining ~70 GB is your entire KV-cache budget, which sets your maximum concurrency at a given context length.

This is the reframe worth carrying into a procurement conversation. Buyers arrive asking for a GPU count. The count is an output, not an input — it falls out of the memory footprint once you know the model, the number format, the context length, and the concurrency target.

Work it in this order:

• Total HBM needed = weights + (KV cache per token × context × concurrency) + headroom.
• GPU count = total HBM ÷ HBM per GPU, rounded up to a power-of-two-friendly sharding layout.
• Node count = GPU count ÷ 8 for HGX-class nodes.

A higher-capacity GPU can therefore reduce your GPU count for the same job — fewer GPUs each holding more, with less cross-GPU traffic and simpler serving. That is the whole argument for stepping from 80 GB to 141 GB to 180 GB to 288 GB parts, and it is a different argument from "the new one is faster".

Not every AI workload needs a stacked-memory accelerator. Smaller models, batch inference at modest concurrency, fine-tuning of small models, vision workloads, and virtual-workstation duty all run well on GDDR-based cards such as the L40S (48 GB GDDR6) or the RTX PRO 6000 Blackwell class (96 GB GDDR7) — parts with far lower bandwidth than an HBM accelerator but far lower cost and power, and no liquid-cooling requirement.

The honest rule: reach for HBM when the model does not fit in a GDDR card’s memory, when you need high concurrency at long context, or when you are training. Otherwise the cheaper part may clear your bar with room to spare.

Once you know your HBM target, the next question is what the GPU actually ships inside — see GPU vs server vs rack, and browse the GPU catalog for the configurations available through Pantheon.