Overfitting and Regularization: Dropout, Weight Decay and Early Stopping

Imagine a student who memorizes the exact answers to every practice problem instead of understanding the underlying method. They'll ace the practice set perfectly but flounder on the real exam, which has slightly different questions. Overfitting is the neural-network version of this: the model's loss on the TRAINING data keeps improving, but its performance on new, unseen data (the validation/test set) stalls or gets WORSE, because the model has started memorizing quirks and noise specific to the training examples rather than learning patterns that generalize. It's especially likely with a big, flexible model and relatively little training data — plenty of capacity to memorize, not enough signal to force genuine learning. Regularization is the umbrella term for techniques that fight this.

Dropout works by randomly 'turning off' (zeroing out) a fraction of neurons — commonly 20-50% — on EVERY training step, forcing the remaining neurons to not rely too heavily on any one specific neuron always being present (since it might be dropped next step). This pushes the network toward learning redundant, more robust representations, similar in spirit to a sports team practicing without always having the same star player available — everyone has to become more well-rounded. Dropout is turned OFF at test/inference time (you use the full network) with a small scaling adjustment.

Weight decay (also called L2 regularization) adds a penalty to the loss function proportional to the SIZE of the weights themselves — the model is now trying to minimize prediction error AND keep its weights small at the same time. Large weights tend to make a model react extremely sensitively to small input changes (a hallmark of memorization/overfitting), so nudging weights toward smaller values encourages smoother, more generalizable functions.

Early stopping is the simplest of the three: track loss on a held-out VALIDATION set (never trained on) throughout training, and stop training as soon as validation loss stops improving — even if training loss is still going down. The gap where training loss keeps falling but validation loss rises is the exact signature of overfitting setting in, and early stopping simply refuses to keep training past that point. In practice, most real training runs combine all three — dropout and weight decay to slow overfitting down, early stopping as the final safety net that catches whatever slips through.

import random

def dropout(activations: list[float], drop_prob: float = 0.3) -> list[float]:
    """Randomly zero out a fraction of activations during training;
    scale survivors up so the expected sum stays the same."""
    keep_prob = 1 - drop_prob
    return [(a / keep_prob) if random.random() > drop_prob else 0.0
            for a in activations]

def weight_decay_penalty(weights: list[float], lambda_: float = 0.01) -> float:
    """L2 penalty added to the loss -- discourages large weights."""
    return lambda_ * sum(w ** 2 for w in weights)

activations = [1.0, 2.0, 3.0, 4.0, 5.0]
print("with dropout(0.3):", [round(a, 2) for a in dropout(activations)])

weights = [0.1, 5.0, -3.2, 0.05]   # one very large weight, likely overfitting signal
print(f"weight decay penalty added to loss: {weight_decay_penalty(weights):.3f}")

def early_stopping(val_losses: list[float], patience: int = 3) -> int | None:
    """Stop when validation loss hasn't improved for `patience` epochs.
    Returns the epoch index to stop at, or None if still improving."""
    best = float("inf")
    epochs_without_improvement = 0
    for epoch, loss in enumerate(val_losses):
        if loss < best:
            best, epochs_without_improvement = loss, 0
        else:
            epochs_without_improvement += 1
        if epochs_without_improvement >= patience:
            return epoch
    return None

val_losses = [0.9, 0.6, 0.4, 0.35, 0.36, 0.38, 0.40]  # stops improving after epoch 3
print("stop training at epoch:", early_stopping(val_losses))