Batch Normalization: Why Deep Networks Are Hard to Train and How It Helps

Key points

• •Deep networks suffer from 'internal covariate shift': each layer's input distribution keeps shifting as earlier layers' weights update during training
• •BatchNorm standardizes each layer's outputs (per mini-batch) to roughly zero mean and unit variance before passing them forward
• •This gives the next layer a stable, familiar input range regardless of how much earlier layers have changed, easing the 'moving target' problem
• •Learnable scale (gamma) and shift (beta) parameters let the network undo the standardization where useful, balancing stability with flexibility
• •Networks with BatchNorm train faster, tolerate higher learning rates, and are less sensitive to weight initialization; at inference it uses running statistics from training instead of a live batch