Tokenization Gotchas: 'cat' vs 'cats', Token Counting and Multilingual Challenges

Understanding HOW tokenizers work (previous two subtopics) explains several surprising, very practical behaviors you'll run into constantly when actually using LLMs.

The first gotcha: to an LLM, 'cat' and 'cats' are not obviously related the way they are to you. Depending on the specific tokenizer and its learned vocabulary, 'cat' might be one token while 'cats' splits into 'cat' + 's' (two tokens) — or, in a different tokenizer, 'cats' might be its OWN single token entirely unrelated in ID-number to 'cat's token ID. The model has to LEARN, purely from training data patterns, that these different token sequences are semantically related — nothing in the tokenization process itself guarantees it. This is why LLMs sometimes behave inconsistently on simple morphological variations (singular/plural, verb tenses) that seem trivial to a human but may correspond to entirely different token sequences to the model.

The second gotcha is token counting: the number of tokens in a piece of text is NOT the number of words, and it's not the number of characters either — the relationship depends entirely on the tokenizer's learned vocabulary and the specific text. Common English words are usually single tokens, but rare words, numbers, code, and punctuation-heavy text often split into MORE tokens than words. This matters enormously in practice: API pricing and context window limits are measured in tokens, not words or characters, so a rough rule of thumb (roughly 4 characters or 0.75 words per token, for English) is useful for estimating cost and length, but it's genuinely just an approximation — always tokenize your actual text if you need an exact count.

The third gotcha is multilingual cost: tokenizers are trained on a corpus, and if that corpus is mostly English (common for many popular tokenizers), the LEARNED vocabulary ends up biased toward English subwords. Text in other languages, especially ones with different scripts (Chinese, Japanese, Korean, Arabic), often gets broken into MANY MORE tokens per unit of actual content than equivalent English text — sometimes 2-5x more. Since cost and context-window usage both scale with token count, this creates a genuine, measurable unfairness: the exact same amount of meaning can cost noticeably more, and eat a bigger share of the context window, in some languages than others. This is a big part of why SentencePiece and larger, more balanced multilingual training corpora matter — they directly reduce this gap.

# Simulating (with a toy vocabulary) the three gotchas: unrelated
# token IDs for related words, token count != word count, and
# a multilingual token-cost comparison.

toy_vocab = {"cat": 501, "cats": 9284, "##s": 12,   # 'cats' has NO relation to 'cat' by ID
             "the": 1, "a": 2, "is": 3, "un": 88, "believ": 4021, "able": 77}

def toy_tokenize(word: str) -> list[str]:
    # Greedy longest-match, same style as the earlier toy tokenizer
    if word in toy_vocab:
        return [word]
    if word == "unbelievable":
        return ["un", "believ", "able"]
    return list(word)  # fallback: characters

print("'cat'  -> token IDs:", [toy_vocab.get(t) for t in toy_tokenize("cat")])
print("'cats' -> token IDs:", [toy_vocab.get(t) for t in toy_tokenize("cats")])
print("-- notice: no numeric relationship between 'cat' and 'cats' IDs --\n")

sentence = "the unbelievable cats"
words = sentence.split()
tokens = [t for w in words for t in toy_tokenize(w)]
print(f"'{sentence}' -> {len(words)} words but {len(tokens)} tokens: {tokens}")

def estimate_tokens(text: str, chars_per_token: float = 4.0) -> int:
    """Rough rule of thumb for English: ~4 chars per token."""
    return round(len(text) / chars_per_token)

english = "How much does this cost?"
print(f"English  ({len(english)} chars): ~{estimate_tokens(english)} tokens (rule of thumb)")
print("Non-Latin-script text often needs MORE tokens per character of content")