GPUs for AI

GPUs for AI

AdvancedTopic4 lessons~4h 5m

Lessons, in order

01 Matrix Multiplication on a GPUAlmost all the time in a neural network is spent multiplying matrices. The textbook three-loop algorithm is correct and hopelessly slow, because it re-reads the same data from far-away … Intermediate 1h 15m 02 Precision and Quantization on HardwareQuantization stores a model's numbers in fewer bits. From the hardware's point of view this is the best trade available, because it improves three limits at once: the model takes less … Intermediate 1h 03 Memory Planning for LLMsAn LLM server's GPU memory holds three things: the weights (fixed), the KV cache (grows with every token of every active conversation), and working space (activations and overhead). The … Intermediate 1h 10m 04 Training vs Inference on a GPUTraining and inference run the same matrix multiplications on the same GPUs, but they stress the hardware in opposite ways. Training is a long, steady, compute-bound batch job that wants … Intermediate 40 min

About this topic

Topics II–IV were about GPUs in general. This topic applies them to the workload that now buys most GPUs: neural networks, and large language models in particular.

#	Lesson	The question it answers
01	Matrix Multiplication on a GPU	How is the single most important operation made fast?
02	Precision and Quantization on Hardware	What do fewer bits buy, in bytes and FLOPs?
03	Memory Planning for LLMs	What fills an AI GPU’s memory, and how many users fit?
04	Training vs Inference on a GPU	Why do the two stress the same hardware so differently?

The sister course, Inference Engineering, takes these ideas up into full serving systems. This topic gives you the hardware-side half.