Self-Attention: How Transformers Understand Context

Self-attention computes: Attention(Q,K,V) = softmax(QKᵀ / sqrt(d_k)) × V. For each token, Q (query) asks "what am I looking for?", K (key) says "what do I contain?", V (value) says "what should I contribute?". The softmax creates attention weights — a probability distribution over all tokens. Each output token is a weighted sum of all value vectors.

Intuition: in the sentence "The bank by the river flooded", attention lets "bank" attend strongly to "river" and "flooded" to understand it means riverbank, not financial institution. RNNs processed left-to-right and would forget context from earlier — attention looks at all tokens simultaneously.

# Simplified attention computation
import numpy as np

def attention(Q, K, V, d_k):
    """Q, K, V shape: [seq_len, d_k]"""
    scores = Q @ K.T / np.sqrt(d_k)  # [seq_len, seq_len]
    weights = softmax(scores, axis=-1)  # attention distribution
    return weights @ V  # [seq_len, d_k]

# For GPT-3: d_k=128 (head_dim), 96 heads, 96 layers
# Each token attends to all previous tokens — O(n²) complexity