Byte Pair Encoding (BPE): Building a Vocabulary from the Corpus
~14 min read
BPE builds its subword vocabulary bottom-up: start with individual characters, then repeatedly merge the most frequent adjacent pair into a new token — the algorithm behind GPT-family tokenizers.
Byte Pair Encoding (BPE): Building a Vocabulary from the Corpus is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •BPE builds its vocabulary bottom-up: start from individual characters, and repeatedly merge the most frequent adjacent pair into a new token
- •The number of merge operations you run is the vocabulary-size knob — more merges capture more (and longer) subwords, including whole common words
- •It discovers useful subwords automatically from corpus statistics, with no human deciding what counts as 'meaningful' in advance
- •Applying a trained BPE tokenizer replays the learned merges in order, so even unseen words get sensibly decomposed into known subword pieces
- •BPE (from Sennrich et al. 2016, adapted from a 1994 compression algorithm) powers the GPT family of tokenizers and remains the most widely used subword method