KV Cache Optimization and PagedAttention
~14 min read
The KV cache stores attention keys and values so the model doesn't recompute them every token — but it eats memory fast. vLLM's PagedAttention manages that memory like an operating system manages RAM.
KV Cache Optimization and PagedAttention is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •The KV cache stores attention keys/values so they're computed once and reused — turning quadratic recompute into a memory lookup
- •Cache size grows with layers x kv_heads x head_dim x sequence_length x batch, and can exceed the model weights themselves
- •Naive contiguous per-request reservation wastes 60-80% of KV memory to fragmentation and over-reservation
- •PagedAttention (vLLM) splits the cache into small fixed-size blocks with a per-request block table — OS-style paging that cuts waste below ~4%
- •Shared prefixes (e.g. a common system prompt) can point to the same physical blocks, saving even more memory via prefix caching