The NVL72 Rack

The idea in one minute#

On an 8-GPU server, GPUs talk over NVLink at 900 GB/s each way; the moment a job needs a ninth GPU, traffic drops to a network card at 50-100 GB/s. That cliff shaped how every large model was split.

The NVL72 moves the cliff. It wires 72 GPUs and 36 CPUs in one liquid-cooled rack so that every GPU reaches every other through NVLink switches in one hop at full speed. To software it is a 72-GPU machine with 13-21 TB of HBM. The price is a rack that draws as much power as a small office building and weighs about 1.4 tonnes. Facts checked on 16 October 2026.

A picture#

How it really works#

What is in the rack#

Why every GPU is one hop from every other#

Each B200 has 18 NVLink 5 links. There are 18 switch chips. Each GPU sends one link to each switch chip, so each chip has 72 GPU ports. Any GPU reaches any other by going up to any switch chip and straight down: one hop, and the full 1.8 TB/s per GPU is available no matter which partner it talks to. The total, 72 × 1.8 TB/s, is the advertised 130 TB/s.

This is a non-blocking, switched fabric. Contrast Google’s TPU pods, which are a direct torus with no switches (see Google TPU): cheaper per link, but distant chips are several hops apart.

Why copper, and why liquid#

• Copper. Inside a rack the longest NVLink run is about a metre. Copper cables there need no optical transceivers, which would have added thousands of lasers and many kilowatts. NVIDIA put the saving at around 20 kW per rack. Beyond a rack, copper cannot carry these speeds, which is why the NVLink domain stops at the rack (until Rubin Ultra’s Kyber racks, lesson 05).
• Liquid. About 120-140 kW in one rack footprint is several times what air can remove. Cold plates sit on every CPU and GPU; a coolant distribution unit exchanges heat with the building’s water loop. Fans remain for power supplies and some boards. 8.c covers the facility side.

What the domain buys you#

The fast domain decides how a model is split (6.b):

• Tensor parallelism needs an all-reduce in every layer and only works inside NVLink. On HGX it stops at 8 GPUs; on NVL72 it can go wider.
• Expert parallelism for mixture-of-experts models sends every token to a few experts on other GPUs and back, in every MoE layer. Spread over 64-72 GPUs inside one NVLink domain, each GPU holds only a few experts and the all-to-all stays on NVLink. That is why trillion- parameter MoE models are served on NVL72 racks.
• KV cache and memory. 13.4 TB of HBM (GB200) or 20.7 TB (GB300) in one domain holds a very large model plus a large shared KV cache.

GB200, GB300, Vera Rubin#

The rack (NVIDIA’s “Oberon” design, built by OEMs from MGX parts) stays the same across three generations, as the table in the figure shows. GB300 swaps in Blackwell Ultra; Vera Rubin NVL72 doubles NVLink to 3.6 TB/s per GPU and brings HBM4.

Code#

Time one MoE layer’s all-to-all (dispatch tokens to experts, then combine results) when the experts are spread across 64 GPUs, either inside one NVL72 or across eight 8-GPU HGX servers.

// alltoall.go — the MoE all-to-all inside an NVL72 vs across HGX servers.
package main

import "fmt"

func main() {
	const (
		hidden      = 7168.0 // model width (DeepSeek-V3-class)
		bytesPerVal = 1.0    // FP8 activations
		topK        = 8.0    // experts per token
		gpus        = 64.0   // expert-parallel group
	)
	nvlink := 900e9 // NVLink 5, bytes/s each way per GPU
	nic := 100e9    // one 800G NIC per GPU, bytes/s each way

	fmt.Println("tokens/GPU   NVL72 (all NVLink)   8 x HGX (8 per node)   slowdown")
	for _, tokens := range []float64{64, 256, 1024, 4096} {
		// each token is sent to topK experts and its results come back: 2 transfers
		bytes := tokens * topK * hidden * bytesPerVal * 2
		inRack := bytes / nvlink

		// on HGX, a uniformly random expert is on the same node with probability 7/63
		local := 7.0 / (gpus - 1)
		tNV := bytes * local / nvlink
		tNIC := bytes * (1 - local) / nic
		hgx := max(tNV, tNIC)
		fmt.Printf("%9.0f   %15.1f µs   %18.1f µs   %7.1fx\n",
			tokens, inRack*1e6, hgx*1e6, hgx/inRack)
	}
	fmt.Println("\nA 61-layer MoE model with ~58 MoE layers pays this ~58 times per forward pass.")
}

The slowdown is roughly the ratio of NVLink to NIC bandwidth, about 8x, on every MoE layer of every step. Overlapping communication with compute hides some of it on HGX, but not 8x. That single number is the business case for the rack.

Remember this#

• NVL72 = 72 GPUs + 36 CPUs, 18 compute trays, 9 switch trays, one NVLink domain.
• Each GPU links once to each of the 18 switch chips: any-to-any, one hop, 1.8 TB/s per GPU.
• Copper inside the rack saves power; it is also why the domain ends at the rack.
• The rack exists for wide tensor and expert parallelism and for one huge pool of HBM.
• ~120-140 kW, liquid cooled, ~1.4 t: the building has to be designed for it.

Try it#

1. Run alltoall.go. Change the NIC to 400G (50 GB/s). What happens to the slowdown?
2. Change topK to 2 (a Mixtral-style model). Does the case for NVL72 weaken or stay the same? Why?
3. Rubin doubles NVLink to 3.6 TB/s per GPU. Recompute the in-rack time at 4,096 tokens per GPU.

Check yourself#

1. Why does each GPU need exactly one link to each of the 18 switch chips?
2. What forces the NVLink domain to end at the edge of the rack?
3. Which parallelism strategy benefits most from NVL72, and why?
4. Name two facility requirements an NVL72 rack imposes that an air-cooled server rack does not.

Sources#

• NVIDIA GB200 NVL72 and GB300 NVL72
• NVIDIA MGX and the NVL72 reference design (OCP contribution)
• Cisco AI rack for Vera Rubin NVL72 datasheet
• DeepSeek-V3 technical report — the expert-parallel all-to-all in practice