The AI Lifecycle on Hardware

The idea in one minute#

“AI hardware” is not one market. A model passes through seven stages, and each one exhausts a different resource first:

1. Data: storage and CPUs, not GPUs.
2. Pretraining: raw tensor FLOPs, the network between thousands of chips, and uptime.
3. Post-training (SFT, preference tuning, reinforcement learning): a mix, because RL spends much of its time generating, which is inference.
4. Fine-tuning: memory capacity, because optimizer state is several times the model.
5. Evaluation: batch throughput.
6. Serving: compute for prefill, memory bandwidth and capacity for decode.
7. Edge: power and memory capacity.

Knowing which resource a stage exhausts tells you which hardware lever to pay for. This lesson walks the stages; the rest of the topic sizes them. Checked on 16 October 2026.

A picture#

How it really works#

Stage by stage#

1. Data. Crawling, deduplication, filtering, tokenisation and mixing run on CPU clusters (Spark, Ray, Dask) over object storage, at petabyte scale. GPUs appear only for quality classifiers, embeddings and synthetic-data generation, which is itself inference. The bottleneck is storage throughput and CPU cores.

2. Pretraining. Next-token prediction over tens of trillions of tokens. The cost is ≈ 6 × parameters × tokens FLOPs, so the levers are tensor FLOPs at a usable precision (BF16, now FP8, with FP4 arriving), the scale-out network that synchronises gradients, and reliability: with 100,000 chips, something fails every few minutes, so checkpoint speed and fast restart matter as much as peak speed. MoE models add heavy all-to-all traffic, which is why they want the largest scale-up domains (NVL72, TPU cubes). Hardware: NVL72 racks, TPU pods, Trainium UltraServers, tens of thousands of chips, tens to hundreds of megawatts.

3. Post-training. Supervised fine-tuning is ordinary training at smaller scale. Preference tuning and reinforcement learning are different: the policy model generates answers (rollouts), a reward is computed, and the trainer updates the weights, which must then be copied back to the generators. Generation is decode, which is memory-bandwidth-bound, and in reasoning- model RL it is often most of the GPU time. So post-training clusters are split into a trainer pool and a rollout pool running an inference engine (vLLM, SGLang), and weight synchronisation between them is a new bottleneck.

4. Fine-tuning. Adapting an open model to a task. Full fine-tuning with Adam needs roughly 16 bytes per parameter before activations (lesson 03), so capacity binds first. LoRA trains small adapter matrices and QLoRA keeps the base model in 4 bits, which brings a 70B fine-tune down to one or two large GPUs and an 8B fine-tune onto a gaming card.

5. Evaluation. Running benchmark suites, LLM judges and red-team prompts: large batches of inference with no latency target. It is the ideal use for spare, older or spot capacity.

6. Serving. Two workloads in one request. Prefill processes the whole prompt in parallel: compute-bound, like training. Decode produces one token at a time per user: each step reads all the weights and the user’s KV cache, so it is bound by memory bandwidth, and the KV cache for many users and long contexts binds capacity. This split is why serving systems now run prefill and decode on separate pools (disaggregated serving), and why decode-specific hardware (SRAM machines, TPU 8i) exists. For a successful model, serving is most of its lifetime compute.

7. Edge. Phones, PCs, cars and robots run models on NPUs or small GPUs within 5-130 W. Memory capacity limits model size; bandwidth limits speed; battery and heat limit everything. INT4/INT8 and FP4 quantisation are the norm.

Training is paid once, inference forever#

Pretraining a model is one large bill. Serving it is a bill per token for as long as it is used. The code below compares them: for a model that is actually used, inference compute overtakes training within months, and because decode runs far below peak efficiency, it overtakes it sooner in GPU-hours than in FLOPs.

Code#

// lifecycle.go — training is one bill; inference is a bill per token, forever.
package main

import "fmt"

func main() {
	const (
		params    = 70e9  // dense model
		trainToks = 15e12 // pretraining tokens
		peakFLOPs = 989e12
		trainMFU  = 0.40 // fraction of peak achieved in pretraining
		serveMFU  = 0.08 // decode-heavy serving achieves far less
	)
	trainFLOP := 6 * params * trainToks
	trainGPUh := trainFLOP / (peakFLOPs * trainMFU) / 3600
	fmt.Printf("pretraining: %.2e FLOP = %.1f M H100-hours\n\n", trainFLOP, trainGPUh/1e6)

	fmt.Println("tokens served/day   serving GPU-hours/day   days until serving > training")
	for _, perDay := range []float64{1e9, 10e9, 100e9, 1e12} {
		flop := 2 * params * perDay // forward pass only: ~2 FLOPs per parameter per token
		gpuh := flop / (peakFLOPs * serveMFU) / 3600
		fmt.Printf("%17.0e   %21.0f   %29.0f\n", perDay, gpuh, trainGPUh/gpuh)
	}
}

At a billion tokens a day, serving takes about 25 years to match the training bill; at 100 billion a day, which a popular model reaches, about three months; at a trillion, nine days. The serveMFU assumption carries this result: batching and speculative decoding raise it, long contexts lower it.

Remember this#

• Seven stages, seven different bottlenecks: storage/CPU, FLOPs+network+uptime, generation+ training, capacity, throughput, prefill FLOPs vs decode bandwidth, power+capacity.
• RL post-training contains inference: size its rollout pool like a serving fleet.
• Fine-tuning is capacity-bound; LoRA and QLoRA move it onto much smaller hardware.
• Serving is two workloads (prefill and decode) and, for a used model, most of the lifetime compute.

Try it#

1. Run lifecycle.go. Raise serveMFU to 0.2 (aggressive batching). How do the break-even days change?
2. Replace the dense 70B with a MoE of 600B total and 37B active parameters (DeepSeek-V3-class): which params goes into the training formula and which into the serving formula?
3. Pick one row of the bottleneck matrix you disagree with and argue for a different score.

Check yourself#

1. Which stage is bound by storage and CPU rather than accelerators?
2. Why does RL post-training need an inference engine?
3. What makes prefill and decode want different hardware?
4. Why does inference overtake training in GPU-hours sooner than in FLOPs?

Sources#

• Scaling laws and the 6ND approximation (Kaplan et al., 2020)
• The Llama 3 herd of models (Meta, 2024) — training scale, failures and checkpointing in practice
• DeepSeek-R1 (2025) — RL with verifiable rewards
• DistServe: disaggregating prefill and decoding (Zhong et al., 2024)
• QLoRA (Dettmers et al., 2023)