Why Tokenization: Computers Need Numbers, Not Text

Every neural network, no matter how sophisticated, ultimately does nothing but arithmetic on numbers — matrix multiplications, additions, the operations from the neural-networks unit. It has no built-in concept of letters or words at all. So before an LLM can process the sentence 'I love pizza,' something has to convert that string of characters into a sequence of numbers. Tokenization is that conversion process: splitting text into pieces (tokens) and mapping each piece to a unique integer ID from a fixed vocabulary.

The obvious first idea is splitting on whitespace — treat every word as one token: 'I', 'love', 'pizza'. This is simple but runs into what's called the vocabulary problem. Natural language has an enormous, essentially unbounded number of distinct words once you count every inflection (cat, cats, cat's), every proper noun, every technical term, every typo, every language mixed together. A word-level vocabulary big enough to cover realistic text would need millions of entries, making the model's first and last layers (which are sized proportional to vocabulary size) enormous. Worse, ANY word not in that fixed vocabulary — a brand-new word, a rare name, a typo — becomes an 'unknown' token the model has never seen and has no way to understand, no matter how similar it is to words it does know.

The opposite extreme is character-level tokenization: treat every individual character as a token. This solves the vocabulary problem beautifully — there are only a few hundred distinct characters in most languages, so the vocabulary stays tiny and NOTHING is ever truly 'unknown.' But it creates a new problem: sequences become very long (the word 'pizza' becomes 5 separate tokens instead of 1), and the model has to work much harder to reconstruct meaning from individual letters — 'understanding' now requires learning that p-i-z-z-a means something, from scratch, purely from co-occurrence patterns, rather than getting 'pizza' as one clean unit for free.

Modern LLMs use SUBWORD tokenization as the practical middle ground: common whole words stay as single tokens ('the', 'pizza'), while rarer or unfamiliar words get split into smaller meaningful pieces ('tokenization' might become 'token' + 'ization'). This keeps the vocabulary manageable, keeps sequences reasonably short, and — crucially — lets the model handle words it's never seen before by decomposing them into familiar sub-pieces instead of giving up entirely. The next three subtopics cover the actual ALGORITHMS (BPE, WordPiece, SentencePiece) that decide exactly how to build that subword vocabulary.

# Comparing word-level, character-level, and subword tokenization
# on the same sentence -- illustrating the trade-offs directly.

sentence = "tokenization helps unfamiliarize rare words"

def word_level_tokenize(text):
    return text.split()

def char_level_tokenize(text):
    return list(text.replace(" ", "_"))  # '_' marks spaces as characters too

def toy_subword_tokenize(text, known_subwords):
    """Greedy longest-match subword split, from a small toy vocabulary --
    a simplified stand-in for what BPE/WordPiece actually learn."""
    tokens = []
    for word in text.split():
        i = 0
        while i < len(word):
            for j in range(len(word), i, -1):     # try longest piece first
                piece = word[i:j]
                if piece in known_subwords:
                    tokens.append(piece)
                    i = j
                    break
            else:
                tokens.append(word[i]); i += 1    # fallback: single character
    return tokens

toy_vocab = {"token", "ization", "help", "s", "un", "familiar", "ize",
             "rare", "word"}

print("word-level: ", word_level_tokenize(sentence), "-> vocab needs EVERY word")
print("char-level: ", char_level_tokenize(sentence)[:15], "... (very long, but no unknowns)")
print("subword:    ", toy_subword_tokenize(sentence, toy_vocab))