Positional Encodings for Long Context: RoPE and ALiBi

Self-attention (previous subtopic) treats input as a SET of tokens by default — it has no inherent notion that token 5 comes before token 6. Without some additional signal, 'the cat sat' and 'sat the cat' would look identical to raw attention. Positional encoding is how Transformers inject order information, and the SPECIFIC method used has a big effect on how well a model generalizes to context lengths longer than it was trained on.

The original Transformer approach used fixed or learned positional embeddings — a distinct vector added to each token representing 'this is position 1,' 'this is position 2,' and so on, up to some maximum trained length. This has an obvious ceiling: if a model was trained with embeddings for positions 1 through 4096, it has literally never seen a 'position 5000' vector, and typically fails to generalize well beyond its trained length.

RoPE (Rotary Position Embedding) takes a fundamentally different approach: instead of ADDING a position vector, it ROTATES each token's query and key vectors by an angle proportional to their position, before computing attention scores. The elegant mathematical property this produces: the attention score between two tokens ends up depending only on their RELATIVE distance (position difference), not their absolute positions. This relative-distance property is what makes RoPE much more amenable to extending context length after training — techniques like 'RoPE scaling' (interpolating or adjusting the rotation frequencies) let a model trained at, say, 4k tokens be adapted to handle much longer sequences with comparatively modest additional training. RoPE is used by LLaMA, Mistral, and many other modern open models.

ALiBi (Attention with Linear Biases) takes a different, arguably simpler route: rather than modifying the query/key vectors at all, it adds a penalty directly to the raw attention SCORES, proportional to how far apart two tokens are — closer tokens get less penalty (attend more freely), farther tokens get a bigger penalty (attend less). This bakes in a helpful inductive bias (nearby context usually matters more) directly into the architecture. The paper's own framing, 'Train Short, Test Long,' captures its key selling point: because the penalty is a simple, smooth function of distance rather than a lookup table bounded by training length, models trained with ALiBi at short sequence lengths have been shown to generalize remarkably well to much longer sequences at inference time, without needing the same kind of length-extension trick RoPE-based models typically apply.

Both approaches solve the SAME underlying problem (generalizing position information beyond training length) with different mechanisms — RoPE via rotating vectors before the attention computation, ALiBi via biasing the attention scores directly afterward — and both are meaningfully better at extending to longer contexts than the original fixed/learned positional embedding approach they replaced.

# A simplified illustration of RoPE's rotation and ALiBi's linear
# distance bias -- the two core mechanisms, in isolation.
import math

def rope_rotate(vector: tuple[float, float], position: int,
               base_freq: float = 10000.0) -> tuple[float, float]:
    """RoPE: rotate a 2D (query/key) vector by an angle proportional
    to its position. The key property: the ANGLE BETWEEN two rotated
    vectors depends only on their position DIFFERENCE."""
    theta = position / base_freq
    x, y = vector
    return (x * math.cos(theta) - y * math.sin(theta),
            x * math.sin(theta) + y * math.cos(theta))

v = (1.0, 0.0)
rotated_pos_5 = rope_rotate(v, position=5)
rotated_pos_105 = rope_rotate(v, position=105)   # same vector, 100 positions later
rotated_pos_5005 = rope_rotate(v, position=5005)  # same RELATIVE distance (100) from 4905
print(f"RoPE-rotated vector at position 5:    {rotated_pos_5}")
print(f"RoPE-rotated vector at position 105:   {rotated_pos_105}")
print("-> what matters for attention is the ANGLE BETWEEN two rotated")
print("   vectors, which depends only on their relative distance, not absolute position\n")

def alibi_bias(query_pos: int, key_pos: int, slope: float = 0.1) -> float:
    """ALiBi: subtract a penalty from the raw attention score,
    proportional to distance -- farther tokens get penalized more."""
    distance = abs(query_pos - key_pos)
    return -slope * distance

raw_score = 2.5   # a hypothetical raw attention score before bias
for key_pos in [10, 50, 500, 5000]:
    biased_score = raw_score + alibi_bias(query_pos=10, key_pos=key_pos)
    print(f"query at pos 10 attending to key at pos {key_pos:5d}: "
          f"biased score = {biased_score:.2f}")