Batch Normalization: Why Deep Networks Are Hard to Train and How It Helps

Training a network with many stacked layers introduces a subtle problem that a shallow network barely notices. Every layer's weights get updated a little on every training step (via backpropagation and gradient descent). But that means layer 5's INPUT — which is layer 4's output — is constantly shifting around as layer 4's weights change underneath it. Layer 5 has to keep re-adapting to a moving target, and this compounds through many layers: by the time you reach layer 20, its inputs might be shifting wildly step to step, which slows training and makes it unstable (this phenomenon is often called 'internal covariate shift'). It's like trying to hit a target that keeps moving every time you adjust your aim.

Batch Normalization (BatchNorm) fixes this by re-standardizing each layer's outputs, for every mini-batch of training examples, before passing them to the next layer. Concretely, for each feature (each neuron's output across the batch), it subtracts the batch's mean and divides by the batch's standard deviation — the same 'z-score' standardization you may have seen in basic statistics, giving each feature roughly zero mean and unit variance within that batch. This means the NEXT layer always receives inputs in a familiar, stable range, no matter how much the previous layer's raw weights have shifted — the moving-target problem is largely neutralized.

After standardizing, BatchNorm adds back two small LEARNABLE parameters — a scale (gamma) and a shift (beta) — so the network can still learn to widen, narrow, or re-center the distribution if that's actually useful for a particular layer, rather than being forced into exactly zero-mean/unit-variance always. This gives the network the STABILITY benefit of standardization while keeping the FLEXIBILITY to undo it where needed.

The practical payoff is substantial: networks with BatchNorm typically train faster (you can often use a higher learning rate safely), are less sensitive to how weights were initialized, and are noticeably more stable overall. It's also worth knowing that at test/inference time, BatchNorm can't use a 'batch' (you might be predicting on just one example), so it instead uses a running average of the mean and standard deviation collected throughout training — a detail that trips up many people implementing it for the first time.

import math

def batch_norm(activations: list[float], gamma: float = 1.0,
               beta: float = 0.0, eps: float = 1e-5) -> list[float]:
    """Standardize a batch of one neuron's activations to zero mean /
    unit variance, then apply a learnable scale (gamma) and shift (beta)."""
    n = len(activations)
    mean = sum(activations) / n
    variance = sum((a - mean) ** 2 for a in activations) / n
    std = math.sqrt(variance + eps)
    normalized = [(a - mean) / std for a in activations]
    return [gamma * z + beta for z in normalized]   # learnable scale/shift

# One neuron's raw outputs across a mini-batch of 5 examples --
# notice the wide, awkward range before normalizing
raw_batch = [120.0, 85.0, 300.0, 95.0, 150.0]
normalized = batch_norm(raw_batch)
print("raw batch:       ", raw_batch)
print("batch-normalized:", [round(v, 3) for v in normalized])

mean_after = sum(normalized) / len(normalized)
print(f"mean after normalization: {mean_after:.6f}  (should be ~0)")
# Real usage: torch.nn.BatchNorm1d(num_features) inserted between layers