Text Tokenization Basics
Learn how text characters are converted into subwords and mapped to unique number IDs for AI processing.
Type something to test subword tokenizer:
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain why models process tokens instead of raw characters or whole words
- •Describe how vocabulary dictionaries map strings to integer IDs
- •Analyze how punctuation and whitespace affect token counts
What is it?
Tokenization is the process of splitting raw text into discrete units (tokens) and mapping each unit to an integer ID from a fixed vocabulary. LLMs do not process text — they process sequences of integers. GPT-4 uses a vocabulary of ~100,000 tokens via the cl100k_base BPE encoding; Llama 3 uses ~128,256 tokens. The token is the atomic unit of both input and output: every character you type is first tokenized, then processed as a matrix of embeddings, and finally the model outputs probability distributions over the vocabulary to predict the next token ID.
Why it exists
Neural networks perform matrix multiplications on fixed-size numerical inputs. They cannot process arbitrary-length strings directly. Tokenization bridges the gap: convert variable-length text → fixed-vocabulary integer sequence → embedding matrix → transformers. The subword approach (BPE/WordPiece) solves the vocabulary explosion problem: character-level means tiny vocab but very long sequences; word-level means huge vocab with out-of-vocabulary problems; subword BPE finds the sweet spot at 32K-128K tokens covering all text efficiently.
Problem it solves
Three problems: (1) Vocabulary explosion — BPE limits vocabulary to 32K-128K tokens while covering all text including rare words, proper nouns, and code by splitting them into subwords. (2) Out-of-vocabulary — since BPE can fall back to individual bytes (byte-level BPE), it handles any Unicode character. (3) Sequence length efficiency — "unbelievable" as one token (7 chars, 1 token) vs character-level (12 chars, 12 tokens) reduces compute by 12×.
Intuition
When you encounter an unfamiliar word like "hyperparametrization," you break it into recognizable pieces: "hyper" + "parameter" + "ization." BPE tokenizers do exactly this — they learn which character sequences appear together frequently in training data and merge them into single tokens.
If you come from Java/Spring Boot: a tokenizer is like a Java serializer (ObjectOutputStream). It converts a complex object (text string) into a byte stream (token IDs) that can be transmitted to the model. The vocabulary is like a class registry — each class (token) has a unique ID. Deserialization is the decoder that converts token IDs back to text. Just as you cannot deserialize bytes from one JVM serializer using another, you cannot use GPT-4's tokenizer with Llama — vocabularies are model-specific.
If you come from React/Frontend: tokenization is like JSON.stringify() for text. You convert a human-readable string into a format the model understands (an array of numbers). The token vocabulary is like an enum of allowed values. When the model generates text, it's like JSON.parse() — converting the number array back to a string. The critical insight for UI developers: "tokens ≠ characters ≠ words" — 1 token ≈ 4 characters for English, but emoji cost 1 token and Chinese characters cost 1-2 tokens each.
Analogy
Tokenization is like Morse code: every letter maps to a pattern of dots and dashes (token ID), and the decoder converts the pattern back to the original letter. But BPE is smarter — common words ("the", "ing", " is") get their own single Morse pattern rather than being spelled out character by character.
Technical explanation
BPE (Byte-Pair Encoding) algorithm:
- Start with vocabulary = set of all bytes (256 tokens)
- Count all adjacent pairs in training corpus
- Merge most frequent pair into new token
- Repeat until vocab reaches target size (e.g., 50,257 for GPT-2)
Example BPE merge sequence: corpus: "low low low lowest" Step 1: ["l","o","w"] most frequent pair → "lo" Step 2: ["lo","w"] most frequent → "low" Step 3: "low" → add "low" to vocab with ID 450
Key tokenizer families:
- tiktoken (OpenAI): cl100k_base (100K vocab, GPT-4), o200k_base (GPT-4o)
- SentencePiece (Google): BPE + WordPiece, used by T5, Llama 2
- HuggingFace fast tokenizers: Rust-based, 100× faster than Python
- Llama 3 tokenizer: 128,256 vocab, byte-level BPE
Token counting matters for:
- Cost: GPT-4 charges per input + output token (not per word or character)
- Context limits: Claude has 200K token limit, not 200K word limit
- Latency: generation speed is measured in tokens/second
Tokens are NOT uniform across languages: "Hello" → 1 token (English) "안녕하세요" → 3 tokens (Korean, ~60% less efficient than English) "print(x)" → 2 tokens (code is very efficient) "🚀" → 1 token (most emoji = 1 token)
Architecture
Tokenizer pipeline:
- Pre-tokenizer: regex split on whitespace/punctuation boundaries
- BPE model: greedy longest-match lookup in vocab dictionary
- Special token injection: <|system|>, <|user|>, <|assistant|>, <|endoftext|>
- Byte-fallback: any byte not in vocab is encoded as <0xNN> hex token
- Decoder: reverse lookup + byte decoding for output generation
Workflow
- Load tokenizer (model-specific vocabulary + merge rules)
- Pre-tokenize: apply regex to split text at word boundaries
- BPE encode: apply merge rules greedily to find longest matching tokens
- Inject special tokens: add BOS/EOS/system/user delimiters per model template
- Return list of integer token IDs
- At generation time: model outputs token IDs → decode back to text
Example
import tiktoken
OpenAI's tokenizer library
enc = tiktoken.get_encoding("cl100k_base") # GPT-4 tokenizer
text = "Tokenization is fundamental to LLMs!" tokens = enc.encode(text) print(f"Text: {text}") print(f"Token IDs: {tokens}") print(f"Token count: {len(tokens)}") print(f"Tokens decoded: {[enc.decode([t]) for t in tokens]}")
Output:
Token IDs: [5257, 2065, 374, 16188, 311, 445, 11237, 82, 0]
Token count: 9
Tokens decoded: ['Token', 'ization', ' is', ' fundamental', ' to', ' L', 'LM', 's', '!']
Cost estimation for production
def estimate_cost(text: str, price_per_million: float = 3.0) -> float: tokens = enc.encode(text) return len(tokens) / 1_000_000 * price_per_million
system_prompt = "You are a helpful assistant..." print(f"System prompt tokens: {len(enc.encode(system_prompt))}") print(f"Cost per 10K requests: ${estimate_cost(system_prompt) * 10_000:.4f}")
Real-world usage
Token counting is critical for production AI systems: (1) Every API call charges by input + output tokens. A 200-token system prompt at 1M requests/month = 200M tokens = $600 with GPT-4. (2) Context window limits are in tokens, not words — a 200K context window holds roughly 150K English words or 100K lines of code. (3) Prompt injection in RAG: chunk size is measured in tokens, and embedding models have token limits (512 tokens for most sentence transformers).
Trade-offs
Larger vocabularies (128K vs 32K) produce shorter sequences (lower compute cost per forward pass) but increase the embedding matrix size and memory. Llama 3 upgraded from 32K to 128K tokens partly to improve code and multilingual efficiency. The embedding layer alone for 128K vocab at 4096 dimensions = 2GB of parameters.
Visual explanation
Tokenization flow: Text: "tokenizing text" ──(Splitter)──> ["token", "izing", " text"] ──(Lookup)──> [1203, 459, 2309]
Advantages
- —
Handles out-of-vocabulary words smoothly by falling back to byte-level tokens
- —
Subwords balance vocabulary size with sequence length efficiently
- —
Code-trained tokenizers are very efficient for Python/JavaScript (common tokens like "def", "import", "return" are single tokens)
Disadvantages
- —
Token boundaries create weird artifacts: "9.11 > 9.9" confuses models because numbers are tokenized non-uniformly. "11" might be one token but "9.11" splits as "9", ".", "11"
- —
Language bias: English text uses ~4 chars/token; Thai/Arabic text uses 1-2 chars/token, costing 2-4× more
- —
Tokenizer mismatch: using tiktoken to count tokens for a Llama model gives wrong counts — always use the model-specific tokenizer
Common mistakes
- —
Assuming 1 token ≈ 1 word. In reality, 1 token ≈ 4 characters for English. "Tokenization" = 3 tokens. Numbers like "1000000" = 1-3 tokens depending on the tokenizer. Always use tiktoken or the model's tokenizer library to get exact counts before calculating costs.
- —
Using the wrong tokenizer for a model. GPT-4 uses cl100k_base. Claude uses a different tokenizer. Llama 3 uses a SentencePiece tokenizer with 128K vocab. Counting tokens with tiktoken for a Llama model will give wrong counts — potentially off by 20-30%, causing silent context overflow.
- —
Not accounting for special tokens in context window calculations. Every chat API call adds BOS/EOS tokens, role delimiters, and template tokens. For multi-turn conversations, the system prompt + all prior messages are re-tokenized on every turn, consuming context window rapidly.
- —
Ignoring tokenization in RAG chunking. Splitting text by "500 characters" does not guarantee "< 512 tokens." Always chunk by tokens using the embedding model's tokenizer. A 500-character chunk of emoji-heavy text could be 200 tokens or 400 tokens depending on content.
- —
Forgetting that token generation is sequential. LLMs generate one token at a time (autoregressive). Generating 100 tokens requires 100 sequential forward passes. This is why latency = time-to-first-token + (tokens_to_generate × ms_per_token). You cannot parallelize output generation.
🎤 Interview questions
Explain how Byte-Pair Encoding (BPE) vocabulary construction works. How are merges prioritized?
📂 Subtopics
Why Tokenization: Computers Need Numbers, Not Text
Neural networks only understand numbers, so text has to be chopped into pieces and mapped to numeric IDs before a model can touch it. That chopping is tokenization, and the piece size you choose creates a fundamental trade-off.
~11 min
Byte Pair Encoding (BPE): Building a Vocabulary from the Corpus
BPE builds its subword vocabulary bottom-up: start with individual characters, then repeatedly merge the most frequent adjacent pair into a new token — the algorithm behind GPT-family tokenizers.
~14 min
WordPiece and SentencePiece: How BERT and T5 Tokenize Differently
WordPiece merges by likelihood gain rather than raw frequency (used by BERT); SentencePiece treats text as a raw byte stream with no whitespace pre-splitting (used by T5 and many multilingual models) — same subword idea, different engineering choices.
~13 min
Tokenization Gotchas: 'cat' vs 'cats', Token Counting and Multilingual Challenges
Tokenization creates real, practical surprises: an LLM sees 'cat' and 'cats' as almost entirely unrelated token sequences, token counts rarely match word or character counts, and non-English text often costs more tokens for the same content.
~12 min