SRAM Machines: Groq and Cerebras

The idea in one minute#

On a GPU, generating one token for one user means reading every weight from HBM. However many FLOPs the chip has, a single user’s tokens per second cannot exceed bandwidth divided by model size (2.c). SRAM machines remove that limit by never keeping the weights in DRAM at all. The weights are spread across the on-chip SRAM of many chips (Groq) or one whole silicon wafer (Cerebras), where they can be read tens of times faster.

The trade is capacity. A Groq chip holds 500 MB; a Cerebras wafer holds 44 GB; an NVIDIA rack holds 20 TB. So these machines are spectacular at per-user speed on models that fit, and need many chips per model. NVIDIA’s own response, licensing Groq’s technology and selling the Groq 3 LPX rack beside its GPUs, shows how much that speed now matters for agents and reasoning models. Facts checked on 16 October 2026.

A picture#

How it really works#

Cerebras: the wafer is the chip#

A normal chip is one rectangle cut from a 300 mm wafer. Cerebras keeps nearly the whole wafer as one chip, the Wafer-Scale Engine:

• WSE-3 (2024): 46,225 mm², 4 trillion transistors, about 900,000 cores, 44 GB of SRAM with about 21 PB/s of memory bandwidth, and about 125 PF of peak AI compute. The CS-3 system around it draws about 23 kW.
• Each core sits next to its own 48 KB of SRAM and a router; a 2D mesh moves data between neighbours. There is no HBM and no cache hierarchy.
• Yield: a wafer always has defects. Spare cores and spare mesh links let the design route around the bad ones, the same idea as the H100’s fused-off SMs at a much larger scale.
• Cooling and packaging are custom: a wafer cannot sit in a normal socket, so the system delivers power and water through the face of the wafer.

Models larger than 44 GB are split across several wafers, layer by layer. For training, Cerebras streams weights from an external memory system (MemoryX) through the wafer, keeping only activations on chip. Its cloud has served 70B-class models at well over 2,000 tokens per second per user, many times a GPU. At Hot Chips 2026 Cerebras presented a rack-scale system design (“Nexus”) for its 2027 CS-5 and said CS-6 will stack DRAM on top of the wafer, an admission that 44 GB is the constraint to fix.

Groq: many small chips, scheduled by a compiler#

Groq took the opposite physical route to the same goal:

• Each LPU (language processing unit) is a normal-sized chip with a large SRAM and no HBM. The LP30 in NVIDIA’s Groq 3 LPX has 500 MB of SRAM at about 150 TB/s per chip.
• A model is laid out across hundreds of chips: each holds a slice of the layers, and activations (a few megabytes per token) flow from chip to chip. A 70B model at FP8 needs about 140 chips.
• The chips are deterministic: no caches, no out-of-order execution, no arbitration. The compiler schedules every unit and every chip-to-chip transfer for every clock in advance, so the run time of a program is known before it runs, and hundreds of chips stay in lock-step without the tail-latency surprises of a dynamic system.
• An LPX rack holds 256 LPUs: 128 GB of SRAM, about 40 PB/s of aggregate SRAM bandwidth, a 640 TB/s scale-up fabric, and 12 TB of DDR5 for larger models and caches. It ships through cloud providers and OEMs in the second half of 2026.

NVIDIA licensed Groq’s technology in December 2025 in a deal reported at $20 billion; GroqCloud continued as a separate service. NVIDIA positions LPX as a decode accelerator beside Rubin: GPUs do prefill and large batches, LPUs generate the latency-critical tokens. The first LP30 lacks NVFP4 and NVLink; NVIDIA has said later parts will add them.

The others#

• SambaNova (SN40L) uses three memory tiers in one system (hundreds of MB of SRAM, 64 GB of HBM and up to 1.5 TB of DDR per socket), so it can hold many models and switch between them quickly, with the hot one in fast memory.
• d-Matrix and similar designs compute inside SRAM arrays (“in-memory compute”), aimed at low-cost, low-latency inference.
• TPU 8i and Maia 200 are a sign of convergence: mainstream inference chips are adding hundreds of MB of SRAM to keep more of decode’s working set on chip (lessons 03 and 04).

When SRAM machines win, and when they do not#

Code#

Chips needed and the per-user decode ceiling for a 70B model at FP8, on a GPU, on LPUs and on a wafer. The SRAM machines’ ceiling is set by the time to stream each chip’s slice plus the chip-to-chip hops, not by HBM bandwidth.

// sram.go — per-user speed: HBM bandwidth vs SRAM spread across chips.
package main

import (
	"fmt"
	"math"
)

func main() {
	const modelGB = 70.0 // 70B parameters at FP8

	// GPU: every token reads all the weights from HBM.
	for _, g := range []struct {
		name string
		tbs  float64
	}{{"B200", 8}, {"Rubin", 22}} {
		fmt.Printf("%-16s 1 GPU        ≤ %5.0f tok/s per user\n", g.name, g.tbs*1000/modelGB)
	}

	// SRAM machines: weights spread over n chips; a token visits every chip in turn.
	type sram struct {
		name      string
		chipGB    float64 // SRAM per chip
		chipTBs   float64 // SRAM bandwidth per chip
		hopMicros float64 // chip-to-chip latency per hop (assumed)
	}
	for _, s := range []sram{
		{"Groq LP30", 0.5, 150, 0.6},
		{"Cerebras WSE-3", 44, 21000, 5},
	} {
		chips := math.Ceil(modelGB / (s.chipGB * 0.9))
		perChip := modelGB / chips / (s.chipTBs * 1000) // seconds to read one chip's slice
		tokenTime := chips*perChip + (chips-1)*s.hopMicros*1e-6
		fmt.Printf("%-16s %4.0f chips   ≤ %5.0f tok/s per user  (memory %.0f µs, hops %.0f µs)\n",
			s.name, chips, 1/tokenTime, chips*perChip*1e6, (chips-1)*s.hopMicros)
	}
	fmt.Println("\nHop latencies are assumptions; real systems also pipeline many users through the chips.")
}

The GPU ceiling is a few hundred tokens per second per user, whatever its FLOPs. On LPUs each chip reads its slice in about 3 µs, but a token visits 156 chips in sequence, so the walk down the chain (plus the hops) sets the ceiling: still about 15x a B200. On two wafers the memory and hop terms almost vanish, and the ceiling the program prints is meaningless; compute, the on-wafer pipeline and software overheads take over, and Cerebras reports thousands of tokens per second, not hundreds of thousands. Real systems also pipeline many users through the chain at once, which is how they reach useful total throughput.

Remember this#

• SRAM machines keep the weights on chip, removing the HBM bandwidth ceiling on per-user decode.
• Cerebras WSE-3: one wafer, 900,000 cores, 44 GB SRAM, 21 PB/s; CS-6 will add stacked DRAM.
• Groq LP30: 500 MB SRAM per chip, compiler-scheduled and deterministic; 256 per LPX rack, sold by NVIDIA beside Rubin since 2026.
• The trade is capacity: many chips per model, and long-context KV caches do not fit well.
• Mainstream inference chips (TPU 8i, Maia 200) are adding SRAM too.

Try it#

1. Run sram.go. Halve the LPU hop latency. Then double it. How sensitive is per-user speed?
2. Change the model to 400B at FP8. How many LP30s and how many wafers? Would one LPX rack hold it without the DDR tier?
3. Add a KV cache of 20 GB that must also live on chip. How many extra LPUs does that take?

Check yourself#

1. Why can no amount of extra FLOPs raise a GPU’s single-user decode speed above bandwidth ÷ model size?
2. How does Cerebras make a wafer-sized chip yield?
3. What does “deterministic, compiler-scheduled” mean on an LPU, and what does it buy?
4. Give two workloads where an SRAM machine is the wrong choice, and why.

Sources#

• Cerebras WSE-3
• Hot Chips 2026: Cerebras lays out the future of wafer-scale AI (Tom’s Hardware)
• NVIDIA Groq 3 LPX
• A closer look at NVIDIA’s Groq-powered LPX racks (The Register, March 2026)
• Groq: Think Fast, a tensor streaming processor (Abts et al., ISCA 2020)
• SambaNova SN40L (arXiv 2405.07518)