Context Window Scaling
Techniques for handling very long contexts in production LLM apps and when to use RAG vs long context
▶📚 Prerequisites(3)
🎓 Learning objectives
- •Explain why context length scaling is quadratically expensive
- •Describe RoPE scaling and how models extend context
- •Know the lost-in-the-middle problem and its implications
- •Choose between long context, RAG, and prompt caching for a given use case
- •Calculate KV cache memory for a model configuration
What is it?
Context window scaling refers to the techniques — better positional encodings (RoPE, ALiBi), sparse or sliding-window attention, and retrieval-augmented approaches — that let LLMs process far more text in a single call than the original Transformer architecture could handle, since raw self-attention cost grows quadratically with sequence length. Modern models now support context windows from 128K tokens (GPT-4o) up to 1 million tokens (Gemini), but bigger isn't always better: cost, latency, and the 'lost in the middle' problem mean knowing when to use long context versus RAG is itself a core production skill.
Why it exists
Real-world documents (codebases, books, legal contracts, conversation histories) easily exceed standard context windows. As context grows, memory and compute costs scale quadratically with standard attention.
Problem it solves
Inputs too long for a single LLM call. High KV cache memory cost for long contexts. The "lost in the middle" problem where models ignore content in the middle of very long prompts.
Intuition
Imagine trying to answer questions about a 500-page book by reading it all at once vs using an index to find relevant pages. Both work, but for different tasks and budgets.
Analogy
Standard context = a whiteboard you can see entirely. Long context = a huge wall of whiteboards — you can technically see all of it but you naturally focus on the edges. RAG = a filing cabinet — you pull only the relevant folders.
Technical explanation
RoPE (Rotary Position Embedding) is the dominant positional encoding scheme. Models extend context by scaling RoPE's base frequency (YaRN, LongRoPE). Sliding window attention processes long sequences in overlapping chunks rather than full attention. KV cache memory = 2 × num_layers × num_heads × head_dim × seq_len × bytes_per_element. For a 70B model with 128K context: ~100GB KV cache alone. Prompt caching (Anthropic, OpenAI) stores KV cache for a fixed prefix — dramatically reduces cost for repeated long-context calls with the same system prompt. Context distillation compresses conversation history into a summary to free up context space.
Architecture
Full attention: O(n²) memory and compute. Sliding window: O(n × window_size). RAG: O(k) where k = retrieved chunks (k << n). Prompt caching: pay full price once, cached portion is ~90% cheaper on repeat calls.
Workflow
Decision framework:
- Is the full document needed for reasoning across all parts? → Long context.
- Is it search/lookup over a large corpus? → RAG.
- Is it a repeated workflow with the same long system prompt? → Prompt caching.
- Is the context growing over a long conversation? → Summarize older turns (context distillation).
Example
Code review bot on a 10K line codebase: RAG retrieves relevant files per question (cheap). Legal contract analysis where clauses reference each other: long context (Claude 200K) needed. Customer support bot with a 50-page FAQ: prompt caching — pay once, reuse KV cache on every query.
Real-world usage
Long context: legal AI, code analysis, book summarization. RAG: enterprise knowledge bases, documentation search. Prompt caching: any app with a large static system prompt (saves 80-90% on that portion).
Trade-offs
Cost vs simplicity: long context is simpler to implement but expensive. RAG is cheaper but adds retrieval complexity and can miss cross-document reasoning.
Visual explanation
Full attention: O(n²) memory Sliding window: O(n × window_size) RAG: O(k) where k = retrieved chunks Prompt cache: pay once for prefix, ~90% cheaper on repeats
Advantages
- —
Long context eliminates chunking complexity
- —
Prompt caching dramatically reduces costs for repeated queries
- —
Sliding window enables processing of arbitrarily long sequences
Disadvantages
- —
Long context is expensive: 1M tokens = $15 on GPT-4o
- —
Lost-in-the-middle degrades quality past ~32K tokens for many models
- —
KV cache requires significant GPU memory
Common mistakes
- —
Using long context for tasks where RAG would be 10x cheaper
- —
Not using prompt caching when system prompt is static
- —
Ignoring lost-in-the-middle — always put the most important content at the start or end of the context
🎤 Interview questions
What is the lost-in-the-middle problem?
How does RoPE scaling extend context length?
When would you choose RAG over stuffing the full document into context?
What is prompt caching and how does it save cost?
Calculate KV cache memory for a given model config.
📂 Subtopics
What Limits Context Windows: Attention Complexity, Memory, and Cost
Self-attention costs grow QUADRATICALLY with sequence length, not linearly — doubling context length roughly quadruples the compute and memory needed for attention alone.
~13 min
Positional Encodings for Long Context: RoPE and ALiBi
A model needs to know token ORDER, not just content. RoPE encodes position via rotation and generalizes to unseen lengths; ALiBi biases attention scores directly by distance — both enable longer context than the original fixed positional embeddings.
~14 min
Long Context Techniques: Sliding Window, Sparse Attention, and Retrieval-Augmented Approaches
Beyond better positional encodings, three architectural strategies directly reduce the O(n^2) attention cost: restrict what each token can see (sliding window), skip most pairs (sparse attention), or avoid full-context attention altogether (retrieval).
~13 min
Practical Implications: Long Context vs. RAG, and Real Model Limits
A large context window doesn't mean you should always use it — cost, latency, and the 'lost in the middle' problem shape when long context genuinely helps versus when RAG is still the better tool.
~13 min