Tokenization: How LLMs Read Text
~15 min read
Before an LLM can process any text, it must convert it into tokens — integer IDs from a fixed vocabulary. Understanding tokenization explains why LLMs behave differently on different languages and why costs vary.
Tokenization converts text into integer token IDs using Byte Pair Encoding (BPE). The tokenizer has a vocabulary of 50K–130K common subwords. "Hello" → [15496], "tokenization" → [5263, 1634] (split into two subwords). English averages 0.75 tokens per word; code and other languages are less efficient (1–3 tokens per word). Cost consequence: Claude at $3/1M input tokens × 10K tokens/doc = $0.03/document.
💻 Code example
import tiktoken # OpenAI tokenizer
enc = tiktoken.get_encoding("cl100k_base") # GPT-4 tokenizer
text = "Hello, how are you today?"
tokens = enc.encode(text)
print(tokens) # [15339, 11, 1268, 527, 499, 3432, 30]
print(len(tokens)) # 7 tokens for 27 characters
# Cost estimate
cost_per_token = 3 / 1_000_000 # $3 per 1M tokens
cost = len(tokens) * cost_per_token
print(f"Cost: ${cost:.6f}")
💬 Deep Dive with AI
Key points
- •BPE tokenizer builds vocabulary from most frequent character pairs
- •1 token ≈ 4 characters in English, less for other languages
- •Token count determines cost — longer inputs cost more
- •Tokenizer must match the model — mixing tokenizers corrupts input