Workstations, PCs and the Edge

The idea in one minute#

Not every AI workload belongs in a data center. Developers fine-tune on a desk, companies keep private data on premises, and robots must think without a network. NVIDIA sells three kinds of hardware for this, and they make opposite bets about memory:

• Cards (RTX 5090, RTX PRO 6000) use fast GDDR7, about 1.8 TB/s, but only 32-96 GB of it. They decode quickly but hold mid-size models.
• Unified-memory machines (DGX Spark, Jetson AGX Thor) give the GPU 128 GB of LPDDR5X shared with the CPU, at only 273 GB/s. They hold big models but decode them slowly.
• DGX Station puts a data-center GB300 superchip under a desk: 784 GB of coherent memory, of which 288 GB is HBM3e.

Pick by the model you need to hold and the speed you need to decode. Facts checked on 16 October 2026.

A picture#

How it really works#

The cards#

• RTX 5090 (GeForce, Blackwell GB202): 32 GB GDDR7 at 1,792 GB/s, 575 W. The fastest single-user decode you can buy for a PC, for models up to ~30B parameters at 4 bits, or larger ones split across cards over PCIe.
• RTX PRO 6000 Blackwell: the same class of die with 96 GB of ECC GDDR7 at 600 W (a 300 W Max-Q variant exists), plus a Server Edition for data-center racks. It holds a 70B model at FP8 on one card. Neither card has NVLink; multi-card setups talk over PCIe, so they suit pipeline or data parallelism better than tensor parallelism.
• L4 (Ada, 24 GB, 72 W) is the data-center cousin: low-profile, passively cooled, cheap per hour, and still a good fit for small models, embeddings and video.

Unified memory: DGX Spark and Jetson Thor#

• DGX Spark is built on the GB10 superchip: a 20-core Arm CPU and a Blackwell GPU sharing 128 GB of LPDDR5X at 273 GB/s, about 1 PF of FP4 with sparsity, in a box that sits on a desk. A ConnectX-7 port links two of them. NVIDIA positions it for models up to ~200B parameters at 4 bits (two linked, ~405B). It is a development machine: the same CUDA stack as the data center, at a few tokens per second on large models.
• Jetson AGX Thor is the robotics module: Blackwell GPU, a 14-core Arm CPU, 128 GB of LPDDR5X, about 2 PF of FP4 (sparse), within a configurable 40-130 W. It runs perception, vision- language-action models and small LLMs on the robot. Its predecessor, Jetson AGX Orin, is still widely deployed.

Because the GPU addresses all of the memory directly, there is no PCIe copy and no “does it fit in VRAM” question below 128 GB. The price is bandwidth: about one-seventh of a GDDR7 card.

DGX Station#

A desk-side machine with a GB300 superchip: a Grace CPU with LPDDR5X and a Blackwell Ultra GPU with 288 GB of HBM3e at about 8 TB/s, joined by NVLink-C2C, for 784 GB of coherent memory and an 800 Gb/s ConnectX-8 NIC. Models that fit in the 288 GB of HBM run at data-center speed; the LPDDR5X holds overflow such as KV cache or optimizer state.

The pattern across all of them#

Code#

Size a local deployment: weights plus KV cache for a given context length, then the decode ceiling (bandwidth over bytes read per token) and the prefill time for a long prompt (compute bound).

// local.go — will this model run on this box, and how fast?
package main

import "fmt"

type Device struct {
	Name     string
	MemGB    float64
	BWGBs    float64
	DenseTFs float64 // dense FP8/FP4-class tensor TFLOP/s actually usable for prefill
}

type Model struct {
	Name                     string
	ParamsB, BitsPerWeight   float64
	Layers, KVHeads, HeadDim float64
}

func main() {
	devs := []Device{
		{"RTX 5090", 32, 1792, 800},
		{"RTX PRO 6000", 96, 1792, 1000},
		{"DGX Spark", 128, 273, 250},
		{"Jetson AGX Thor", 128, 273, 500},
		{"DGX Station (HBM part)", 288, 8000, 5000},
	}
	m := Model{"70B, 4-bit, GQA (8 KV heads)", 70, 4.5, 80, 8, 128}
	const ctx, prompt = 32768.0, 8000.0

	weights := m.ParamsB * m.BitsPerWeight / 8 // GB
	// K and V, per layer, per KV head, FP8 cache: 1 byte each
	kvPerToken := 2 * m.Layers * m.KVHeads * m.HeadDim * 1 / 1e9 // GB
	kv := kvPerToken * ctx
	fmt.Printf("%s: weights %.1f GB, KV cache at %.0fk context %.1f GB\n\n", m.Name, weights, ctx/1024, kv)
	fmt.Println("device                    fits   decode ≤ tok/s   8k-token prefill ≥ s")
	for _, d := range devs {
		need := weights + kv + 2 // + runtime overhead
		if need > d.MemGB {
			fmt.Printf("%-24s  no (needs %.0f GB)\n", d.Name, need)
			continue
		}
		decode := d.BWGBs / (weights + kv) // each token reads weights and the whole cache
		prefill := 2 * m.ParamsB * 1e9 * prompt / (d.DenseTFs * 1e12)
		fmt.Printf("%-24s  yes  %14.0f   %18.1f\n", d.Name, decode, prefill)
	}
}

The TFLOP/s column is a rough usable dense figure, not the marketing number: NVIDIA quotes FP4 with sparsity, which a 4-bit dense model cannot use in full. Notice how the Spark and the Thor hold the model with room for a long context but decode at single-digit tokens per second, and how long prefill takes on the small boxes: a long prompt is compute-bound, and these are small GPUs.

Remember this#

• Cards: fast GDDR7, 32-96 GB. Unified-memory boxes: slow LPDDR5X, 128 GB. DGX Station: both.
• Capacity decides what fits; bandwidth decides decode speed; compute decides prefill time.
• RTX PRO 6000 (96 GB) is the largest-memory NVIDIA card and holds a 70B model at FP8.
• DGX Spark is a development machine on the data-center software stack, not a serving box.
• Jetson AGX Thor brings Blackwell to robots at 40-130 W.

Try it#

1. Run local.go. Shorten the context to 4k. Which devices change answer?
2. Add an Apple Mac Studio (512 GB, 819 GB/s, ~100 usable TFLOP/s). Where does it land?
3. Change the model to 8B at BF16 (16 bits per weight, 32 layers, 8 KV heads). Which device decodes it fastest, and by how much?

Check yourself#

1. Why does DGX Spark hold a larger model than an RTX 5090 but decode it more slowly?
2. Why do two RTX PRO 6000 cards suit pipeline parallelism better than tensor parallelism?
3. What part of a DGX Station’s 784 GB runs at data-center speed?
4. Which resource limits prefill on a small device, and which limits decode?

Sources#

• NVIDIA DGX Spark and DGX Station
• NVIDIA RTX PRO 6000 Blackwell
• GeForce RTX 5090
• Introducing NVIDIA Jetson Thor (NVIDIA technical blog)
• NVIDIA L4