WordPiece and SentencePiece: How BERT and T5 Tokenize Differently

BPE isn't the only subword algorithm in wide use — two close cousins solve the same underlying problem (build a manageable subword vocabulary) with different design choices, and knowing which model uses which explains some tokenization quirks you'll encounter.

WordPiece, used by BERT and its many descendants, is extremely close to BPE with one key difference in HOW it picks which pair to merge next. Plain BPE always merges the single MOST FREQUENT adjacent pair. WordPiece instead merges whichever pair gives the biggest increase in the LIKELIHOOD of the training corpus under a simple language model — roughly, it favors merging a pair when the merged token is much more predictable together than the two pieces are independently, rather than purely how often the pair occurs. In practice this produces broadly similar-looking vocabularies to BPE, but WordPiece is also known for a distinctive OUTPUT convention: it prefixes non-word-initial subword pieces with '##' (so 'tokenization' might become 'token' + '##ization'), making it visually obvious, just by looking at the tokens, which pieces start a new word and which continue one.

SentencePiece, used by T5, ALBERT, XLNet and many multilingual models, solves a different problem: both BPE and WordPiece, as originally described, assume you can pre-split text on whitespace before tokenizing WITHIN each word. That assumption breaks for languages like Japanese, Chinese, or Thai that don't use spaces between words at all. SentencePiece's fix is to skip whitespace pre-splitting entirely: it treats the input as one raw, continuous stream of Unicode characters (or even raw bytes), TREATING SPACES AS ORDINARY CHARACTERS THEMSELVES (typically marked with a special symbol, '▁', a visible underscore-like marker), and then runs a BPE-style or unigram-language-model-style algorithm directly on that raw stream. Because spaces are just another character to the algorithm, SentencePiece is 'language agnostic' — the exact same algorithm handles space-separated languages and non-space-separated languages uniformly, which is exactly why it's the standard choice for multilingual models.

The practical takeaway: all three (BPE, WordPiece, SentencePiece) are solving the SAME core problem from the previous subtopic (manageable vocabulary, no unknown-word failures) with different merge-selection rules and different assumptions about pre-splitting — the specific choice mostly matters for multilingual robustness and for interpreting the raw token strings a model shows you.

# Illustrating the two key differences: WordPiece's '##' continuation
# marker, and SentencePiece's whitespace-as-character '_' marker.

def bpe_style_tokens(word: str, pieces: list[str]) -> list[str]:
    """Plain BPE-style output: no marker distinguishing word-continuation pieces."""
    return pieces  # e.g. ['token', 'ization']

def wordpiece_style_tokens(pieces: list[str]) -> list[str]:
    """WordPiece marks non-word-initial pieces with '##' so you can tell,
    just from the token string, which pieces continue a word."""
    return [pieces[0]] + [f"##{p}" for p in pieces[1:]]

def sentencepiece_style_tokens(text: str, pieces_per_word: list[list[str]]) -> list[str]:
    """SentencePiece marks the START of each word (i.e. where a space
    was) with '_', treating whitespace as an ordinary character."""
    out = []
    for pieces in pieces_per_word:
        out.append("_" + pieces[0])   # '_' stands in for SentencePiece's '\u2581'
        out.extend(pieces[1:])
    return out

word_pieces = ["token", "ization"]
print("BPE-style:           ", bpe_style_tokens("tokenization", word_pieces))
print("WordPiece-style:     ", wordpiece_style_tokens(word_pieces))
print("SentencePiece-style: ", sentencepiece_style_tokens(
    "deep tokenization", [["deep"], ["token", "ization"]]))

# Real usage:
#   from transformers import BertTokenizer, T5Tokenizer
#   BertTokenizer.from_pretrained('bert-base-uncased').tokenize('tokenization')
#   -> ['token', '##ization']