Beam Search: Keeping Top-k Sequences Alive, and Its Trade-offs vs. Greedy
~15 min read
Beam search approximates true whole-sequence maximization by keeping the top-k partial sequences alive at every step, rather than committing to one path like greedy decoding — at the cost of k times the compute.
Beam Search: Keeping Top-k Sequences Alive, and Its Trade-offs vs. Greedy is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Beam search keeps the top-k partial sequences alive at every step, instead of committing to one path like greedy decoding
- •This approximates true whole-sequence maximization, which requires knowing future conditionals that aren't available step-by-step
- •A beam that starts with a slightly less-probable token can still win overall if its full continuation scores higher
- •Costs roughly k times the compute of greedy decoding, since k candidate continuations run in parallel
- •Widely used where correctness of the whole sequence matters more than creativity, e.g. machine translation — less suited to chat or storytelling