Bits-per-Character: Normalizing Cross-Entropy Across Different Tokenizers

The probability-basics prerequisite material already defines perplexity as 2 raised to the cross-entropy loss in bits, and explains it as 'how confused the model is, on average.' That definition has a subtle trap once you try to use it to compare TWO DIFFERENT models: perplexity is computed PER TOKEN, and different models often use completely different tokenizers with different vocabulary sizes and different average token lengths. A model with a large vocabulary that packs more text into each token will naturally have a LOWER per-token perplexity than an otherwise-identical model with a smaller vocabulary — not because it's genuinely better at modeling language, but simply because each of its 'guesses' covers less linguistic ground per token. Comparing raw perplexity numbers across models with different tokenizers is genuinely comparing apples to oranges.

Bits-per-character (BPC) fixes this by normalizing to a unit that doesn't depend on tokenizer choice at all: instead of measuring bits of surprise per TOKEN, measure bits of surprise per CHARACTER of the underlying text. The conversion is straightforward: take the model's total cross-entropy loss (in bits) for a piece of text, and divide by the number of CHARACTERS that text contains (not the number of tokens). Because every model, regardless of its tokenizer, is ultimately being asked to predict the SAME underlying character sequence, BPC gives you a fair, apples-to-apples comparison — a model with a large-vocabulary tokenizer no longer gets an artificial advantage just from packing more characters into fewer tokens.

The practical takeaway for evaluation work: whenever you're comparing language modeling quality ACROSS models that might use different tokenizers (which is the common case comparing, say, an open-source model against a proprietary API model), reach for bits-per-character rather than raw per-token perplexity or cross-entropy. When you're tracking ONE model's progress over training (same tokenizer throughout), raw per-token perplexity is perfectly fine, since the tokenizer is held constant and the comparison is already fair. This distinction — same-tokenizer tracking versus cross-tokenizer comparison — is exactly the kind of practical evaluation subtlety that sits one level above the pure definitions in the probability-basics material, and it's a common, genuine source of misleading benchmark comparisons in published language-model research when it's overlooked.

# Demonstrating why raw per-token perplexity misleads across
# different tokenizers, and how bits-per-character (BPC) fixes it.
import math

def cross_entropy_bits_per_token(token_probs: list[float]) -> float:
    """Average -log2(p) across the tokens the model assigned to the
    actual text -- this is what raw perplexity is built from."""
    return sum(-math.log2(p) for p in token_probs) / len(token_probs)

def perplexity_from_bpt(bits_per_token: float) -> float:
    return 2 ** bits_per_token

def bits_per_character(total_bits: float, num_characters: int) -> float:
    """Normalize to a TOKENIZER-INDEPENDENT unit: bits per character
    of the underlying text, not bits per (tokenizer-specific) token."""
    return total_bits / num_characters

text = "the quick brown fox"   # 19 characters

# Model A: a SMALL-vocabulary tokenizer -- more tokens, each covering
# fewer characters (e.g. splits into ~8 tokens for this text)
model_a_token_probs = [0.6, 0.5, 0.55, 0.6, 0.5, 0.45, 0.6, 0.55]
bpt_a = cross_entropy_bits_per_token(model_a_token_probs)
total_bits_a = bpt_a * len(model_a_token_probs)

# Model B: a LARGE-vocabulary tokenizer -- fewer, longer tokens for
# the SAME text (e.g. only 4 tokens), so naturally lower bits/TOKEN
# even with comparable per-token confidence
model_b_token_probs = [0.35, 0.30, 0.32, 0.33]
bpt_b = cross_entropy_bits_per_token(model_b_token_probs)
total_bits_b = bpt_b * len(model_b_token_probs)

print(f"Model A: {bpt_a:.3f} bits/token -> perplexity {perplexity_from_bpt(bpt_a):.2f}")
print(f"Model B: {bpt_b:.3f} bits/token -> perplexity {perplexity_from_bpt(bpt_b):.2f}")
print("-> Model B LOOKS much better on raw per-token perplexity\n")

bpc_a = bits_per_character(total_bits_a, len(text))
bpc_b = bits_per_character(total_bits_b, len(text))
print(f"Model A: {bpc_a:.3f} bits/character (fair, tokenizer-independent)")
print(f"Model B: {bpc_b:.3f} bits/character (fair, tokenizer-independent)")
print("-> Once normalized to bits/character, the comparison is fair --")
print("   the gap may shrink or even reverse once tokenizer effects are removed")