Pidoku
KV Cache Management

KV Cache Management

IntermediateTopic5 lessons~4h 10m

Lessons, in order

01 Blocks and the PoolAll the KV cache memory a vLLM engine will ever use is allocated once, at startup, and cut into fixed-size blocks. From then on, "memory management" means moving small Python objects between … Intermediate 55 min 02 Prefix CachingIn a causal language model, the keys and values computed for token i depend only on tokens 0 to i. Two prompts that begin with the same tokens therefore have identical KV cache contents for … Intermediate 55 min 03 Allocating Slots: One Call, TracedEvery time the scheduler wants to give a request some tokens, it makes one call: kv_cache_manager.allocate_slots(request, num_new_tokens, …). The answer is either a list of new block IDs or … Intermediate 45 min 04 Sizing the Cache: From Gigabytes to BlocksThe number of blocks in the pool is fixed at startup and never changes. It is the result of one subtraction and one division: take the share of the GPU you allowed vLLM to use, subtract what … Intermediate 50 min 05 More Than One Kind of CacheSo far every layer of the model needed the same thing from the cache: room for every token, forever. Many current models break that assumption. Some layers look only at the most recent … Intermediate 45 min

About this topic

The KV cache is where a serving engine’s memory goes, and how it is managed decides how many requests fit on a GPU. vLLM allocates all of it once, cuts it into blocks, and from then on only moves block numbers between a free queue, the requests that own them and a table of hashes. This topic reads that machinery: the pool, the cache built on top of it, the single function through which every allocation passes, and the arithmetic that sizes it.

#LessonThe question it answers
01Blocks and the PoolWhat is a block, where does a freed block go, and why is that order an eviction policy?
02Prefix CachingHow does vLLM recognise a prompt it has seen before, and why is my hit rate low?
03Allocating Slots: One Call, TracedWhat exactly is checked when the scheduler asks for memory?
04Sizing the Cache: From Gigabytes to BlocksHow many tokens fit on my GPU, and which flags change that?
05More Than One Kind of CacheWhat changes for models whose layers do not all need a full-length cache?

For the ideas without the implementation, see KV Cache, KV Cache Math, Paged Attention and Prefix Caching.

esc

↑↓ navigate↵ openesc close

drag to pan · scroll to zoom