What Limits Context Windows: Attention Complexity, Memory, and Cost
~13 min read
Self-attention costs grow QUADRATICALLY with sequence length, not linearly — doubling context length roughly quadruples the compute and memory needed for attention alone.
What Limits Context Windows: Attention Complexity, Memory, and Cost is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Self-attention computes a score between every pair of tokens, so cost scales with sequence length SQUARED (O(n^2)), not linearly
- •Doubling context length roughly quadruples attention compute — this is the core structural reason context windows can't scale for free
- •The KV cache (from llm-optimization) must store every token's keys/values for the whole context, making very long contexts memory-bound in GPU VRAM
- •Longer prompts take meaningfully longer to process (the 'prefill' phase) and cost more, which is part of why API pricing scales with token count
- •Real long-context capability requires specific engineering work (covered in the next two subtopics) to manage this quadratic cost, not just more raw compute