Tokenization Gotchas: 'cat' vs 'cats', Token Counting and Multilingual Challenges
~12 min read
Tokenization creates real, practical surprises: an LLM sees 'cat' and 'cats' as almost entirely unrelated token sequences, token counts rarely match word or character counts, and non-English text often costs more tokens for the same content.
Tokenization Gotchas: 'cat' vs 'cats', Token Counting and Multilingual Challenges is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •'cat' and 'cats' can tokenize into entirely different, numerically unrelated token sequences — the model must learn their relationship purely from data
- •Token count is neither word count nor character count — it depends on the tokenizer's vocabulary and the specific text (rough rule of thumb: ~4 chars/token for English)
- •API pricing and context-window limits are measured in tokens, so always tokenize the actual text if you need an exact count, not word/character counts
- •Tokenizers trained on mostly-English corpora often need noticeably more tokens (sometimes 2-5x) for equivalent content in other scripts/languages
- •This multilingual token-cost gap is a real, measurable fairness issue — better multilingual corpora and algorithms like SentencePiece help close it