The Memory Map

The idea in one minute#

For AI, and for language models above all, two numbers describe an accelerator better than its FLOPs: how much memory it has (what fits) and how fast it can read that memory (how fast it decodes). Plot every accelerator on those two axes, on log scales, and the whole industry falls into three clusters:

• SRAM machines (Groq, Cerebras): tiny capacity, enormous bandwidth.
• HBM accelerators (NVIDIA, AMD, TPU, Trainium, Maia): 100-450 GB at 3-23 TB/s.
• LPDDR and GDDR devices (Apple, DGX Spark, Jetson, gaming cards): large or modest capacity at well under 2 TB/s.

The diagonal lines on the chart add a third number: the time to read the whole memory once, the floor on one decode step when the weights fill it. Checked on 16 October 2026.

A picture#

How it really works#

Reading the map#

• Moving right means larger models or more KV cache fit on one device.
• Moving up means each token of a memory-bound decode is faster.
• The diagonals are capacity ÷ bandwidth. A device on the “10 ms” line takes 10 ms to read all of its memory, so a model that fills it decodes at no more than 100 tokens per second per user. Data-center GPUs sit between about 10 and 40 ms; the SRAM machines below 3 µs per chip; LPDDR devices at 400-600 ms.

Three clusters, three products#

Within the HBM cluster, generations move up more than right: Rubin roughly triples bandwidth at the same capacity as B300, while AMD’s MI455X moves both, to 432 GB.

What the map hides#

• Batching. The map describes one user. With many users, the weights are read once per step for all of them, and compute or KV-cache bandwidth becomes the limit instead.
• Mixture of experts. Only the active experts are read per token, so a MoE model decodes as if it were much smaller than its capacity footprint.
• Scale-up. Eight GPUs on NVLink act like one device with 8x the capacity and bandwidth, for a model split across them. An NVL72 is a point far off the right edge of the chart.
• Price and power. Neither axis is cost. Divide by price before deciding.

Code#

Classify devices by their read-all time, and list which can serve a 70B FP8 model to a single user at a given speed target.

// memmap.go — capacity, bandwidth, and the time to read everything once.
package main

import (
	"fmt"
	"sort"
)

type Dev struct {
	Name     string
	GB, TBs  float64
	ScaleOut int // devices that can be joined on a fast fabric for one model
}

func main() {
	devs := []Dev{
		{"Groq LP30", 0.5, 150, 256},
		{"Cerebras WSE-3", 44, 21000, 4},
		{"H100", 80, 3.35, 8},
		{"H200", 141, 4.8, 8},
		{"B200", 192, 8, 72},
		{"Rubin", 288, 22, 72},
		{"MI355X", 288, 8, 8},
		{"MI455X", 432, 19.6, 72},
		{"TPU Ironwood", 192, 7.37, 64},
		{"Trainium3", 144, 4.9, 144},
		{"Apple M3 Ultra", 512, 0.819, 1},
		{"DGX Spark", 128, 0.273, 2},
		{"RTX 5090", 32, 1.79, 1},
	}
	sort.Slice(devs, func(i, j int) bool { return devs[i].GB/devs[i].TBs < devs[j].GB/devs[j].TBs })
	fmt.Println("device            read-all (ms)   class")
	for _, d := range devs {
		ms := d.GB / d.TBs // GB ÷ TB/s = ms
		class := "HBM"
		switch {
		case ms < 0.1:
			class = "SRAM"
		case ms > 100 || d.TBs < 2:
			class = "LPDDR/GDDR"
		}
		fmt.Printf("%-16s %14.3f   %s\n", d.Name, ms, class)
	}

	const modelGB, target = 70.0, 100.0 // 70B FP8, tokens/s per user
	fmt.Printf("\nserve %.0f GB at ≥ %.0f tok/s for one user (memory bound, tensor-parallel group):\n", modelGB, target)
	for _, d := range devs {
		for n := 1; n <= d.ScaleOut; n *= 2 {
			cap, bw := d.GB*float64(n)*0.85, d.TBs*float64(n)
			if cap >= modelGB && bw*1000/modelGB >= target {
				fmt.Printf("  %-16s × %-3d  ≤ %6.0f tok/s\n", d.Name, n, bw*1000/modelGB)
				break
			}
		}
	}
}

Every HBM accelerator reaches 100 tokens per second on a 70B model with a group of one to four; the LPDDR devices never do, however many you have, because they cannot be joined on a fast fabric; the SRAM machines need many chips and clear the target easily. Ignore the size of their printed speeds: the program adds up bandwidth across chips as if one user could use all of it at once, but a token walks a chain of chips one after another (see SRAM machines for a better model).

Remember this#

• Capacity says what fits; bandwidth says how fast a single user decodes; their ratio is the floor on one decode step.
• Three clusters: SRAM (µs), HBM (~10-40 ms), LPDDR/GDDR (tens to hundreds of ms).
• New HBM generations move mostly up (bandwidth); AMD moves right (capacity) too.
• The map is single-user; batching, MoE and scale-up domains change the picture.

Try it#

1. Run memmap.go. Raise the target to 300 tokens per second. Which devices still qualify, and at what group size?
2. Add the Maia 200 (216 GB, 7 TB/s) and the Intel GPU D-series (160 GB of LPDDR5X; look up its bandwidth). Which cluster is the Intel part in?
3. Where would an NVL72 rack sit if drawn as a single point on the chart?

Check yourself#

1. What does a point on the “10 ms” diagonal tell you?
2. Why can’t four DGX Sparks serve a 70B model as fast as one H200?
3. Name two effects the single-user map leaves out.

Sources#

• Vendor data sheets for each device, as cited in the NVIDIA hardware line and Beyond NVIDIA
• Apple Mac Studio technical specifications
• Roofline: an insightful visual performance model (Williams et al., 2009)