Why Tokenization: Computers Need Numbers, Not Text
~11 min read
Neural networks only understand numbers, so text has to be chopped into pieces and mapped to numeric IDs before a model can touch it. That chopping is tokenization, and the piece size you choose creates a fundamental trade-off.
Why Tokenization: Computers Need Numbers, Not Text is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Neural networks only compute on numbers, so text must be converted to a sequence of integer token IDs before any model can process it
- •Word-level tokenization needs a huge vocabulary (millions of words) and still fails on any word it's never seen (out-of-vocabulary/'unknown' tokens)
- •Character-level tokenization keeps the vocabulary tiny and has no unknowns, but produces very long sequences and harder-to-learn meaning
- •Subword tokenization is the practical middle ground: common words stay whole, rare words split into meaningful pieces — manageable vocabulary, reasonable length, handles unseen words
- •The choice of tokenization scheme directly trades off vocabulary size, sequence length, and how gracefully the model handles unfamiliar words