Seven Levels of Scale

The idea in one minute#

AI hardware comes in nested sizes, each roughly ten times the power of the one inside it:

1. Die: one piece of silicon.
2. Package: what we call “a GPU” (one or two dies plus HBM).
3. Node: a server, typically 8 GPUs on NVLink.
4. Rack: an NVL72 or TPU cube, the largest fast (scale-up) domain.
5. Pod: a few to a few dozen racks on one network fabric.
6. Cluster: tens of thousands of accelerators in one building.
7. Campus: several buildings, hundreds of thousands to a million chips, approaching gigawatts.

Every workload has a smallest level that can hold it (memory) and a smallest level that can finish it in time (compute). Pick the larger of the two, and try to keep each tightly coupled part of the work inside one fast domain. Checked on 16 October 2026.

A picture#

How it really works#

What changes at each step#

The most important boundary is the edge of the scale-up domain: inside it, GPUs share memory semantics at NVLink speed; outside it, they exchange messages at a tenth of the bandwidth or less. On HGX servers that edge is at 8 GPUs; on NVL72 at 72; on a TPU pod, the torus extends to 9,216 chips but with more hops.

Matching parallelism to levels#

A frontier pretraining run nests all of them: tensor and expert inside each rack, pipeline across a few racks, data parallelism across the cluster.

Power is the real limit#

Each level up is about 10x the power. A single NVL72 rack (120-200 kW) is already more than most older data-center halls can feed per rack; a cluster of 100,000 GPUs needs 150 MW or more; the campuses announced by several labs for 2026-28 are measured in gigawatts. Long before money runs out, grid connections, transformers and cooling water set the scale (8.c).

Code#

For each workload, find the smallest level whose memory holds it and the smallest that finishes it within a deadline, and report the larger.

// levels.go — the smallest level of scale that holds a job and finishes it in time.
package main

import (
	"fmt"
	"math"
)

type Level struct {
	Name string
	GPUs float64
}

func main() {
	levels := []Level{{"package (1 GPU)", 1}, {"node (8)", 8}, {"rack (72)", 72},
		{"pod (576)", 576}, {"cluster (16k)", 16384}, {"cluster (100k)", 100000}}
	const (
		memPerGPU  = 288.0   // GB, B300-class
		flopPerGPU = 2.25e15 // dense BF16, B200-class
		mfu        = 0.4
	)
	jobs := []struct {
		Name     string
		MemGB    float64 // total memory the job needs
		FLOP     float64 // total work (0 for serving: sized by memory)
		Deadline float64 // days
	}{
		{"serve 8B BF16, 32 users × 8k", 16 + 32*8192*131072/1e9, 0, 0},
		{"serve 70B FP8, 64 users × 32k", 70 + 64*32768*163840/1e9, 0, 0},
		{"serve 1T MoE FP8 + 2 TB KV", 1000 + 2000, 0, 0},
		{"full fine-tune 70B, 2B tokens", 70e9 * 16 / 1e9 * 1.3, 6 * 70e9 * 2e9, 2},
		{"pretrain 70B on 15T tokens", 70e9 * 16 / 1e9 * 1.5, 6 * 70e9 * 15e12, 60},
		{"pretrain 1T MoE (40B active), 30T", 1e12 * 16 / 1e9 * 1.5, 6 * 40e9 * 30e12, 90},
	}
	fmt.Println("job                                  needs GPUs (mem)  (time)   smallest level")
	for _, j := range jobs {
		byMem := math.Ceil(j.MemGB / (memPerGPU * 0.85))
		byTime := 0.0
		if j.FLOP > 0 {
			byTime = math.Ceil(j.FLOP / (flopPerGPU * mfu * j.Deadline * 86400))
		}
		need := math.Max(byMem, byTime)
		lvl := "beyond one campus"
		for _, l := range levels {
			if need <= l.GPUs {
				lvl = l.Name
				break
			}
		}
		fmt.Printf("%-36s %10.0f %9.0f   %s\n", j.Name, byMem, byTime, lvl)
	}
}

The KV-cache terms use 131,072 bytes per token for the 8B model (BF16, 32 layers, 8 KV heads) and 163,840 for the 70B (FP8, 80 layers). Notice which column decides each row: serving is sized by memory, training by the deadline. Both pretraining runs need over a thousand GPUs to meet their deadlines, although their weights and optimizer state would fit in one or two racks.

Remember this#

• Seven levels: die, package, node, rack, pod, cluster, campus; each about 10x the power.
• The edge of the scale-up domain (8 GPUs on HGX, 72 on NVL72) is the boundary that matters most.
• Put tensor and expert parallelism inside the domain; pipeline and data parallelism across it.
• Serving is usually sized by memory, training by compute and a deadline.
• At the top, power and buildings, not chips, set the limit.

Try it#

1. Run levels.go. Halve the pretraining deadline for the 70B model. Which level does it need now?
2. Change memPerGPU to 80 (H100). Which serving jobs move up a level?
3. A 405B dense model at FP8 must serve 128 users at 16k context. Add it as a row (126 layers, 8 KV heads, head dimension 128).

Check yourself#

1. Which boundary between levels matters most, and why?
2. Why does expert parallelism belong inside a rack?
3. Why is a training job usually sized by its deadline rather than its memory?
4. What limits the top two levels more than chip supply does?

Sources#

• The Llama 3 herd of models (Meta, 2024) — 4D parallelism on 16k GPUs
• Megatron-LM: efficient large-scale training (Narayanan et al., 2021)
• NVIDIA DGX SuperPOD with GB200 reference architecture
• Google Cloud TPU7x documentation