Bits-per-Character: Normalizing Cross-Entropy Across Different Tokenizers
~13 min read
Comparing two models' perplexity directly is misleading if they use different tokenizers — bits-per-character (BPC) normalizes cross-entropy to a tokenizer-independent unit, making cross-model comparison fair.
Bits-per-Character: Normalizing Cross-Entropy Across Different Tokenizers is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Perplexity/cross-entropy is computed PER TOKEN, so models with different tokenizers (different vocab sizes, different average token lengths) aren't directly comparable on raw per-token numbers
- •A model with a large vocabulary packs more characters per token, which artificially lowers its per-token perplexity without genuinely better language modeling
- •Bits-per-character (BPC) normalizes cross-entropy to bits of surprise per CHARACTER of underlying text, removing the tokenizer-dependence entirely
- •Use BPC when comparing language modeling quality ACROSS models with different tokenizers; raw per-token perplexity is fine for tracking ONE model's progress over training (tokenizer held constant)
- •This tokenizer-fairness issue is a common, genuine source of misleading cross-model perplexity comparisons when overlooked in published benchmarks