Networking: the Other Half

The idea in one minute#

A large GPU cluster has four separate networks, each built for a different job:

1. Scale-up: NVLink inside a server or an NVL72 rack. GPUs read and write each other’s memory at 0.9-1.8 TB/s each.
2. Scale-out (the “back-end”): one 400-800 Gb/s NIC per GPU, wired into a fabric of leaf and spine switches, joining thousands of GPUs for training and large-scale serving.
3. Front-end: ordinary data-center Ethernet for storage, data loading, checkpoints and users.
4. Out-of-band management: a slow, separate network to the baseboard controllers, so a broken fabric can still be repaired.

Each step down this list is roughly ten times less bandwidth per GPU, and distributed training and serving are laid out to match. NVIDIA sells all of it, which is a large part of its advantage. Facts checked on 16 October 2026.

A picture#

How it really works#

Scale-up: NVLink#

Covered in lessons 03 and 04. Its defining property is memory semantics: a GPU issues loads and stores to another GPU’s HBM as if it were its own, with no network stack in between. That is what makes per-layer tensor parallelism and expert all-to-alls cheap enough to do inside every forward pass. NVIDIA now also licenses NVLink to other chip makers (“NVLink Fusion”), so a custom CPU or accelerator can join an NVLink domain.

Scale-out: one NIC per GPU, in rails#

Each GPU gets its own NIC (ConnectX-7 at 400 Gb/s, ConnectX-8 at 800 Gb/s, ConnectX-9 at 1.6 Tb/s with Rubin). With GPUDirect RDMA the NIC reads and writes GPU memory directly; the CPU is not on the data path.

The wiring is rail-optimised: GPU 0 of every server connects to leaf switch 0 (“rail 0”), GPU 1 to rail 1, and so on. Collective libraries (NCCL) arrange their traffic so most of it flows between GPUs of the same index, which are then one switch hop apart. Leaves connect to spines in a non-blocking Clos (fat-tree), two tiers for a few thousand GPUs, three for tens of thousands.

Two fabrics compete for this job:

Both carry 800 Gb/s per port today. The industry trend is toward Ethernet, with UEC as the multi-vendor answer and NVIDIA’s Spectrum-X as its own; for the largest NVIDIA-only training clusters InfiniBand remains common.

Front-end and storage#

Data loading, checkpoints and user traffic go over a separate Ethernet network, often through a BlueField DPU: a NIC with its own Arm cores that runs storage, security and tenant isolation off the host CPU. With GPUDirect Storage, a parallel file system (WEKA, VAST, DDN, Lustre) can move data straight into GPU memory. Checkpoints are the stress test: thousands of GPUs writing terabytes at once, every few minutes to hours.

Optics: where the power goes next#

Every 800G link between racks needs a pluggable optical transceiver at each end, each drawing roughly 15 W, and a large cluster has hundreds of thousands of them. Co-packaged optics put the lasers’ modulators on the switch package itself, cutting power per port and failure points. NVIDIA ships it in its Quantum-X and Spectrum-X Photonics switches and in Rubin’s Spectrum-6. Scale-across networking (Spectrum-XGS) stretches one training job across data centers kilometres apart.

The ratio that shapes everything#

Tensor and expert parallelism live inside NVLink; pipeline and data parallelism cross the NICs, because they send less, less often (6.b).

Code#

Time one gradient all-reduce of a 70B-parameter model (140 GB in BF16) on a single 8-GPU node, in an NVL72, and across 128 nodes using the standard hierarchical scheme: reduce inside each node over NVLink, then across nodes over the NICs, with all 8 rails working in parallel.

// allreduce.go — why gradients are reduced inside NVLink first.
package main

import "fmt"

// ring all-reduce: each GPU sends and receives 2(n-1)/n of the data.
func ring(bytes, n, bwPerGPU float64) float64 { return 2 * (n - 1) / n * bytes / bwPerGPU }

func main() {
	const grads = 140e9 // 70B parameters × 2 bytes
	nvlink, nic := 900e9, 100e9

	fmt.Printf("8 GPUs, NVLink only:            %6.2f s\n", ring(grads, 8, nvlink))
	fmt.Printf("72 GPUs, one NVL72:             %6.2f s\n", ring(grads, 72, nvlink))
	fmt.Printf("1,024 GPUs, flat ring on NICs:  %6.2f s\n", ring(grads, 1024, nic))

	// hierarchical: reduce-scatter in the node, all-reduce 1/8 shards across nodes on
	// 8 rails at once, all-gather in the node.
	nodes := 128.0
	inNode := ring(grads, 8, nvlink)
	across := ring(grads/8, nodes, nic)
	fmt.Printf("1,024 GPUs, hierarchical:       %6.2f s  (%.2f s NVLink + %.2f s NICs)\n",
		inNode+across, inNode, across)
	fmt.Println("\nWith data parallelism this happens every step; overlap with the backward pass hides part of it.")
}

The flat ring over NICs takes 2.8 s; doing the same reduction hierarchically takes 0.62 s, about 4.5x faster, because the bulk of the data never leaves NVLink. Even then, the NIC half costs more than the NVLink half although each NIC carries only an eighth of the data. In practice FSDP/ZeRO-style sharding and overlap with the backward pass hide much of this, but the bandwidth ratio sets the floor.

Remember this#

• Four networks: scale-up (NVLink), scale-out (NICs and switches), front-end (Ethernet, storage, users), management.
• Scale-out is one NIC per GPU, wired in rails, with GPUDirect RDMA.
• InfiniBand and RDMA Ethernet both work; Ethernet is gaining with Spectrum-X and UEC.
• NVLink offers about 9-18x the bandwidth of a NIC per GPU; place parallelism accordingly.
• Optics are a growing share of cluster power; co-packaged optics are the response.

Try it#

1. Run allreduce.go. Change the NIC to 1.6 Tb/s (200 GB/s, ConnectX-9). How much faster is the hierarchical all-reduce?
2. Put 72 GPUs per node (NVL72 racks) instead of 8. Recompute the hierarchical time for 1,008 GPUs (14 racks).
3. On a multi-GPU machine, run nvidia-smi topo -m. Which GPU pairs are on NVLink, and which NIC sits closest to each GPU?

Check yourself#

1. What is rail-optimised wiring, and what does it save?
2. Why is NVLink called a memory-semantic fabric, and why does that matter for tensor parallelism?
3. What does a DPU do that a NIC does not?
4. Why does the management network need to be separate?

Sources#

• NVIDIA Spectrum-X Ethernet and Quantum-X800 InfiniBand
• NVIDIA ConnectX-8 SuperNIC
• Ultra Ethernet Consortium specification 1.0
• NCCL documentation: rings, trees and topology