WordPiece and SentencePiece: How BERT and T5 Tokenize Differently
~13 min read
WordPiece merges by likelihood gain rather than raw frequency (used by BERT); SentencePiece treats text as a raw byte stream with no whitespace pre-splitting (used by T5 and many multilingual models) — same subword idea, different engineering choices.
WordPiece and SentencePiece: How BERT and T5 Tokenize Differently is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •WordPiece (BERT) merges by likelihood gain to the corpus, not raw pair frequency like BPE, and marks continuation pieces with a '##' prefix
- •SentencePiece (T5, ALBERT, multilingual models) skips whitespace pre-splitting entirely, treating spaces as ordinary characters marked with '▁'
- •This makes SentencePiece language-agnostic — the same algorithm handles languages without spaces (Japanese, Chinese, Thai) as easily as English
- •All three algorithms (BPE, WordPiece, SentencePiece) solve the same core problem — manageable vocabulary, no unknown-word failures — with different merge rules
- •The choice matters mainly for multilingual robustness and for reading raw tokenizer output (spotting '##' or '▁' tells you which convention is in play)