Positional Encodings for Long Context: RoPE and ALiBi
~14 min read
A model needs to know token ORDER, not just content. RoPE encodes position via rotation and generalizes to unseen lengths; ALiBi biases attention scores directly by distance — both enable longer context than the original fixed positional embeddings.
Positional Encodings for Long Context: RoPE and ALiBi is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Fixed/learned positional embeddings assign a distinct vector per position up to a trained maximum, and generalize poorly beyond that length
- •RoPE rotates query/key vectors by an angle proportional to position — the resulting attention score depends only on RELATIVE distance, enabling techniques like RoPE scaling to extend context length
- •ALiBi adds a distance-proportional penalty directly to raw attention scores instead — closer tokens attend more freely, farther tokens are penalized more
- •ALiBi's 'Train Short, Test Long' framing means models trained on short sequences generalize remarkably well to much longer ones at inference, without extra length-extension steps
- •Both RoPE (used by LLaMA, Mistral, and others) and ALiBi solve the same generalization problem with different mechanisms — rotate-before-attention vs bias-after-attention