How to Size a GPU Cluster

Sizing runs in one direction, and skipping a link produces a number you will have to throw away:

workload → memory footprint → GPU count → node count → network → power → cooling → floor space

Most sizing conversations start in the middle ("we want 64 H200s") and work outward, which is how buildouts end up with hardware their building cannot feed. Start at the left. The GPU count is an output.

The step that surprises almost everyone is the transition from GPU nameplate power to facility load. Adding up GPU TDPs gets you well under half of what the room actually draws.

Write down what the cluster is for. The two cases size very differently.

Training or fine-tuning. What model size, on how much data, in what wall-clock time? Training is compute- and communication-heavy, which pushes you toward current-generation GPUs and a serious scale-out fabric — and it runs at sustained near-peak power for days, which matters for the facility.

Inference / serving. What model, at what number format, with what context length, at what concurrency, and against what latency target? Serving is usually memory-bound rather than compute-bound, so the sizing lever is memory capacity and bandwidth.

Be specific about context length and concurrency. They quietly dominate the memory number, and they are the two inputs most often left as "we will figure it out later".

Total memory is the sum of three things:

• Weights = parameters × bytes per parameter. A 70-billion-parameter model needs roughly 140 GB at FP16/BF16, or roughly 70 GB at FP8.
• KV cache = per-token cost × context length × concurrent sequences. For a 70B-class model at 16-bit this runs around 0.33 MB per token — roughly 40 GB for a single 128,000-token sequence, before you multiply by concurrency.
• Headroom for activations, framework allocations and fragmentation. A plan that fits exactly does not fit.

Divide the total by the HBM capacity of the GPU you are considering, then round up to a layout that shards cleanly. This is where a higher-capacity part earns its keep: the same job on 288 GB GPUs needs fewer of them than on 80 GB GPUs, with less cross-GPU traffic. What is HBM and why it matters works the arithmetic in detail.

For training, add optimiser states and gradients — with common optimisers this multiplies the weight footprint several times over before any activation memory, which is why training clusters dwarf serving clusters for the same model.

GPU count converts to hardware simply: divide by eight for HGX-class nodes, or size in whole racks if you are buying rack-scale.

Then decide the network, because past a couple of nodes the fabric stops being an accessory. How many NICs per node? Large training clusters commonly run one high-speed NIC per GPU on the east-west fabric, plus separate ports for storage and management — that ratio appears on spec sheets for a reason. InfiniBand or Ethernet? Both are deployed at scale in 2026; see InfiniBand vs Ethernet for AI.

Size storage here too. A cluster that cannot keep its GPUs fed is an expensive idle asset, and checkpointing a large training job writes a great deal, quickly.

This is the step almost every online sizing guide skips, and it is the one that decides whether the project is feasible.

Start from full node power, not GPU TDP. An 8-GPU Blackwell node whose GPUs sum to roughly 8 kW of nameplate draws substantially more at the wall once host CPUs, system memory, NVMe, fans and power-supply losses are included — NVIDIA quotes a maximum of about 14.3 kW for its own 8-GPU B200 system, against 10.2 kW for the equivalent Hopper-generation system. Then add the layers above the node, and then apply your facility’s PUE.

The layers, and what each one adds:

Take 64 Blackwell-class GPUs — eight 8-GPU nodes.

• GPU nameplate: 64 × ~1,000 W ≈ 64 kW. This is the number people quote.
• Node level: 8 nodes at roughly 14 kW each ≈ 114 kW. Already close to double.
• Plus network, storage and management: call it ~125–130 kW of IT load.
• At the meter: apply PUE. At a well-run 1.3 that is roughly 165 kW; at the industry-wide average — the Uptime Institute’s 2025 survey put the weighted average annual PUE at 1.54 — closer to 200 kW.

So 64 kW of GPU nameplate becomes roughly 165–200 kW of grid draw: about 2.5–3× the naive figure. Use that multiple as a sanity check on any plan you are handed. It is approximate, but it is the right order of magnitude.

Scale it and you see why AI buildouts become power projects. A thousand Blackwell-class GPUs — 125 nodes — lands around 1.8 MW of IT load and roughly 2.3–2.8 MW at the meter. Ten thousand GPUs is a mid-sized industrial electrical load. At that point the binding constraint stops being GPU allocation and becomes energy: an available interconnection on a timeline that matches your hardware delivery. Utility interconnection queues in many regions run years, which is why on-site and behind-the-meter generation has moved from a niche to a default consideration — see what is behind-the-meter power and why gas turbines are sold out.

The sequencing lesson: confirm power before you confirm silicon. Both timelines are long, and the one you cannot compress is usually the electrical one.

Power density decides the physical build.

Air cooling is generally practical to around 30 kW per rack, and up to roughly 50 kW with hot-aisle containment or rear-door heat exchangers. Above that, direct-to-chip liquid cooling becomes the norm — and rack-scale systems are liquid-only, with a GB300 NVL72 drawing on the order of 120 kW in a single cabinet.

That sets your floor plan. Eight air-cooled 8-GPU nodes at ~14 kW each will not share one conventional rack: at 30 kW per rack you need roughly three racks, at 15 kW per rack closer to eight. The same GPU count in a liquid-cooled rack-scale form factor occupies far less floor but demands a facility that can deliver the cooling and the density. The cooling decision is unpacked in HGX H100 air vs liquid cooling, the form-factor decision in GB300 NVL72 rack vs HGX nodes.

Also check floor loading, cable routing, service clearance and the delivery path. An integrated rack that will not fit through the door is a real failure mode.

When you have a target shape, tell us the workload, the timeline and what your facility can deliver today, and we will size the configuration against it. Browse the GPU catalog or the wider hardware overview for what is available through Pantheon.