Performance

Performance

IntermediateTopic5 lessons~4h 55m

Lessons, in order

01 The Roofline ModelAny operation needs two things from the GPU: arithmetic (FLOPs) and data (bytes moved to and from memory). The GPU supplies each at a maximum rate. Whichever runs out first sets the speed. Intermediate 1h 10m 02 Memory Access PatternsThe 32 threads of a warp issue their memory requests together. If their 32 addresses are neighbours, the GPU fetches them as one or two wide transactions. If the addresses are scattered, it … Intermediate 1h 03 Occupancy and DivergenceAn SM hides memory waits by switching between many resident warps. Occupancy is how many warps a kernel actually keeps resident, as a fraction of the SM's maximum. Each thread needs … Intermediate 55 min 04 Launch Overhead and FusionEvery kernel launch has a fixed cost of a few microseconds — the host prepares arguments, the driver validates them, a command crosses PCIe. For a kernel that runs for milliseconds, that is … Intermediate 50 min 05 Measuring a GPUYou cannot improve what you have not measured, and GPU metrics are easy to misread. The most famous one — "GPU utilization" — does not mean what almost everyone assumes. This lesson covers … Intermediate 1h

About this topic

A GPU can be a thousand times faster than a CPU or barely faster at all. The difference is almost never “better arithmetic”. It is whether the program respects four constraints: memory bandwidth, memory access patterns, how well the SMs are kept busy, and per-launch overhead.

#	Lesson	The question it answers
01	The Roofline Model	What is the fastest this operation could possibly run?
02	Memory Access Patterns	Why does the order of memory access change speed 10x?
03	Occupancy and Divergence	Are the SMs actually busy?
04	Launch Overhead and Fusion	Why are many small kernels slow?
05	Measuring a GPU	What do the tools tell you — and what do they hide?

When you finish you can look at an operation, predict its speed from a data sheet, measure it, and explain the gap.