Loss Functions: MSE, Cross-Entropy, and Why We Minimize Loss

A freshly initialized neural network's weights are random numbers, so its first predictions are essentially garbage. Training is the process of nudging those weights so predictions get better — but 'better' needs a precise, numeric definition, and that's exactly what a loss function provides: one single number that says how wrong the network's output was on a given example, where a smaller number means a better prediction. Training a neural network, end to end, is simply: compute the loss, then adjust weights to make the loss smaller. That's it — everything else (backpropagation, optimizers) is machinery for doing that adjustment efficiently.

The right loss function depends on what kind of answer you're predicting. For NUMERIC predictions — predicting a house price, a temperature, any continuous number — Mean Squared Error (MSE) is the standard choice: take the difference between the predicted value and the true value, square it (so negative and positive errors both count as 'bad,' and big errors are punished disproportionately more than small ones), and average across all examples. MSE = average of (predicted - actual)^2. A prediction that's off by 10 contributes 100 to the average; a prediction off by 1 contributes only 1 — the squaring makes big mistakes hurt far more than small ones.

For CLASSIFICATION — predicting which category something belongs to, like the next-token prediction from the probability-basics unit — cross-entropy loss is standard. As covered in that unit's information-theory subtopic, cross-entropy measures how well the model's predicted probability distribution matches the true answer (which is 100% probability on the correct class, 0% on everything else). It's minimized exactly when the model puts all its confidence on the right answer, and it punishes confident WRONG predictions especially harshly (predicting 99% probability on the wrong class produces a huge loss) — which is exactly the behavior you want to discourage during training.

The reason we minimize loss rather than, say, maximize accuracy directly, is technical but important: loss functions like MSE and cross-entropy are smooth and differentiable — you can compute a gradient (a direction of steepest improvement) at every point — while raw accuracy is a jagged step function that gives no useful direction to nudge weights in. That smoothness is exactly what the next subtopic, backpropagation, depends on.

import math

def mse_loss(predictions: list[float], targets: list[float]) -> float:
    """For NUMERIC predictions: average squared error.
    Big mistakes are punished disproportionately (squared)."""
    n = len(predictions)
    return sum((p - t) ** 2 for p, t in zip(predictions, targets)) / n

def cross_entropy_loss(predicted_probs: list[float], true_class_idx: int) -> float:
    """For CLASSIFICATION: -log(probability the model assigned to the
    TRUE class). Confident wrong answers are punished heavily."""
    p_true = predicted_probs[true_class_idx]
    return -math.log(max(p_true, 1e-12))  # clip to avoid log(0)

# MSE example: predicting house prices (in $100k)
predicted_prices = [3.2, 4.9, 2.1]
actual_prices    = [3.0, 5.0, 4.0]     # last prediction is way off
print(f"MSE loss: {mse_loss(predicted_prices, actual_prices):.3f}")

# Cross-entropy example: classifying an image as cat(0)/dog(1)/bird(2)
confident_and_correct = [0.95, 0.03, 0.02]   # true class = 0 (cat)
confident_and_wrong   = [0.02, 0.95, 0.03]   # true class = 0 (cat), but predicted dog
print(f"loss (confident + correct): {cross_entropy_loss(confident_and_correct, 0):.3f}")
print(f"loss (confident + WRONG):    {cross_entropy_loss(confident_and_wrong, 0):.3f}")
# The wrong-but-confident prediction gets a MUCH higher loss