GPU vs Server vs Rack: What You Actually Buy

There are four layers, and buyers who skip one get surprised at quote time:

• Accelerator — a single GPU. In the data center this is usually an SXM module, a bare board that bolts onto a baseboard. It has no chassis, no power supply, and no way to stand alone.
• Node (server) — typically eight accelerators on one NVIDIA HGX baseboard, plus host CPUs, system memory, NVMe storage, network cards, power supplies and cooling, in a chassis built and warrantied by an OEM. This is the unit you actually buy.
• Rack — a 42U or 48U cabinet holding some number of nodes, top-of-rack switching, and power distribution. Or, in a rack-scale system, a single integrated product.
• Cluster — many racks stitched together by a scale-out network, shared storage, and a scheduler.

The common mistake is treating layer one as the purchase. It is not.

Data-center SXM accelerators are not sold as loose parts you assemble. Eight of them are mounted at the factory onto an HGX baseboard, which also carries the NVSwitch chips that wire the GPUs to each other. That baseboard is a component NVIDIA supplies to server manufacturers — it is not a retail item, and it does nothing without a host platform around it.

That host platform is not a rounding error. A single 8-GPU node also contains:

• Two host CPUs and a large pool of system memory, which stage data, run the training loop’s Python, and feed the GPUs.
• Local NVMe storage, for checkpoints, datasets and scratch.
• Network cards — commonly one high-speed NIC per GPU for the east-west GPU fabric, plus separate ports for storage and management.
• Power and thermals — multiple redundant power supplies and either a dense air-cooling path or a liquid-cooled cold-plate loop.

So the thing you buy is "an OEM 8-GPU node built on the HGX B200 baseboard, in this cooling configuration, with this CPU and this NIC choice". That is why quotes are configuration-specific, and why the same GPU shows up in several different products. See HGX vs DGX for who integrates it and what that changes.

What each layer is, and what you can actually purchase at that level.

This is the distinction that most often gets garbled, and it matters for every sizing decision downstream.

Inside a node, the eight GPUs talk over NVLink through NVSwitch chips on the baseboard. This is a direct, memory-semantic interconnect with very high per-GPU bandwidth — fourth-generation NVLink runs at 900 GB/s per GPU on Hopper, fifth-generation at 1.8 TB/s per GPU on Blackwell. All eight GPUs behave as one tightly coupled group. This is called scale-up.

Between nodes, GPUs talk over a network — InfiniBand or Ethernet — through the node’s NICs, leaf switches, and spine switches. Per-GPU bandwidth here is an order of magnitude lower than NVLink and latency is higher. This is scale-out, and it is why cluster performance depends so heavily on network design. See InfiniBand vs Ethernet for AI.

The practical rule: work that fits inside one node’s NVLink domain is cheap; work that crosses node boundaries costs network. Model-parallel sharding is deliberately laid out to stay inside the NVLink domain where possible.

A rack-scale system such as the GB200 NVL72 or GB300 NVL72 does something conceptually different: it extends the NVLink domain beyond the node to cover the whole rack.

Instead of nine separate 8-GPU islands connected by a network, an NVL72 rack wires 72 GPUs into a single NVLink domain — around 130 TB/s of all-to-all in-rack bandwidth — alongside 36 Grace CPUs, in-rack NVLink switch trays, liquid cooling, and its own power distribution. Functionally, the rack behaves like one very large accelerator with one very large pool of memory, rather than a small network of servers.

That is why rack-scale is the answer for the largest single-domain training runs and for serving models too big to shard comfortably across networked nodes. It also moves the purchase boundary up a layer: you buy the rack, factory-integrated and burned in, not the nodes inside it. The trade is liquid cooling and rack power density around 120 kW — a facility question, not a silicon question. The full form-factor decision is in GB300 NVL72 rack vs HGX nodes.

Above the rack sits everything that makes many racks act like one machine:

• Scale-out fabric — leaf and spine switches, optics and cabling, usually laid out rail-optimized so each GPU’s traffic has a predictable path.
• Storage — a parallel or high-throughput filesystem, because a training job that cannot keep the GPUs fed is an expensive idle asset.
• Management and head nodes — provisioning, telemetry, the job scheduler.
• Power and cooling plant — the part that decides how much of the above your building can actually run.

None of this is optional, and none of it appears on a GPU spec sheet. When people say a cluster "cost more than the GPUs", this is where the difference went. How to size a GPU cluster walks the arithmetic.

Three things follow directly.

Specify at the node level, not the GPU level. "Eight B200s" is not a purchase order. "An HGX B200 8-GPU node from this OEM, liquid-cooled, with this NIC configuration" is.

The integrator matters. The same HGX baseboard ships inside servers from Dell, Lenovo, Supermicro, GIGABYTE and HPE, with different chassis heights, cooling options, host CPUs, service terms and lead times. Availability often differs more than specification does.

Decide node vs rack-scale early. It drives cooling, rack power density, floor loading and the whole facility conversation — and it is expensive to change late.

Browse the GPU catalog to see how the same silicon is packaged by different integrators, or start from the hardware overview.