Long Context Techniques: Sliding Window, Sparse Attention, and Retrieval-Augmented Approaches
~13 min read
Beyond better positional encodings, three architectural strategies directly reduce the O(n^2) attention cost: restrict what each token can see (sliding window), skip most pairs (sparse attention), or avoid full-context attention altogether (retrieval).
Long Context Techniques: Sliding Window, Sparse Attention, and Retrieval-Augmented Approaches is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Sliding window attention caps each token's attention to a fixed nearby window, changing cost from O(n^2) to O(n*w) — linear in sequence length
- •The tradeoff: tokens lose direct access to far-away context in a single layer, though stacking layers can still propagate information across hops (as in Mistral's design)
- •Sparse attention computes only a carefully chosen SUBSET of pairs (local + a few global tokens), preserving some long-range access while still skipping most of the n^2 pairs
- •Retrieval-augmented approaches (RAG) sidestep the cost differently: keep the actual context short by retrieving only relevant chunks, rather than making full attention cheaper
- •These are complementary strategies — reduce attention cost per token (sliding/sparse) vs. reduce how much content ever enters the context window (retrieval)