Greedy Decoding: Always Pick the Highest-Probability Token, and Why It Fails

Once an LLM has produced a probability distribution over its vocabulary for the next token, something still has to actually pick one. Greedy decoding is the simplest possible answer: at every single step, take the token with the highest probability, append it, and repeat (autoregress) for the next position.

It's appealing for obvious reasons — it's deterministic (the same input always produces the same output), it's fast (no need to explore alternatives), and it's trivial to implement. But greedy decoding has a well-known, fundamental flaw: it only ever optimizes for the single most immediate next token, never the quality of the whole sequence. This narrow, myopic view is exactly what makes it prone to repetitive sentences and loops — once the model commits to a locally-optimal but generically 'safe' token, that choice constrains everything after it, and there's no mechanism to notice or recover if the overall sequence starts sounding stuck or repetitive.

A concrete way to see this: if a model has just generated 'the cat sat on the', greedy decoding might repeatedly favor 'the' again in a similar context later in the sequence, because 'the' is a very high-probability token in general — even when a less common but more contextually apt word would make for better, less repetitive text. Because greedy decoding has zero randomness and zero lookahead, it has no way to break out of these local traps.

In practice, greedy decoding is mostly used where determinism and speed genuinely matter more than output quality or diversity — for instance, quick internal testing, latency-critical applications where every millisecond counts, or tasks with an extremely narrow, well-defined correct answer where creativity would actively hurt (simple classification-style outputs). For anything involving open-ended generation — chat, stories, summaries, code with any stylistic freedom — greedy decoding is rarely the right default, which is exactly why the other three strategies in this topic exist.

import numpy as np

def greedy_decode(model_step_fn, initial_tokens: list[int], max_new_tokens: int = 20) -> list[int]:
    """model_step_fn(tokens) -> probability distribution over the vocabulary
    for the NEXT token, given the tokens so far."""
    tokens = list(initial_tokens)
    for _ in range(max_new_tokens):
        probs = model_step_fn(tokens)
        next_token = int(np.argmax(probs))  # ALWAYS the single highest-probability token
        tokens.append(next_token)
        if next_token == EOS_TOKEN_ID:
            break
    return tokens

# Illustrating the repetition problem: a toy model that keeps favoring
# the same high-frequency token regardless of context length
def toy_repetitive_model(tokens: list[int]) -> np.ndarray:
    vocab_size = 5
    probs = np.array([0.5, 0.2, 0.15, 0.1, 0.05])  # token 0 ("the") always dominant
    return probs

EOS_TOKEN_ID = -1
print(greedy_decode(toy_repetitive_model, [1], max_new_tokens=10))
# -> repeatedly picks token 0 every single step, producing a degenerate loop