Deep Learning & Gradient Descent
Learn loss functions, gradients, training iterations, and how models adjust weights to reduce errors.
Initialize Random Position
The algorithm starts training at a randomized set of model coordinates (weights).
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Explain what a Loss Function (like MSE) measures
- •Understand how Gradient Descent steps weights down the error curve
- •Define Learning Rate and explain the risk of overshooting
What is it?
Deep learning trains neural networks with multiple hidden layers to learn hierarchical representations directly from raw data. Each layer learns features of increasing abstraction — pixels → edges → shapes → objects. The key mechanic is gradient descent: the network computes a scalar loss, backpropagates gradients through every weight via the chain rule, then nudges each weight a tiny step opposite to its gradient. Repeat millions of times and the loss converges.
Why it exists
Initially, networks were too hard to train. Gradient descent provided the mathematical engine to update millions of weights automatically.
Problem it solves
Enables automatic parameter search: instead of hand-tuning weights, the model learns them from raw data.
Intuition
Imagine walking down a foggy mountain path to find the valley (lowest error). You cannot see the bottom, but you can feel the slope under your feet and step downward.
Analogy
Tuning weights is like adjusting dials on a radio. The loss function is the volume of static noise; gradient descent tells you which way to turn the dials to get clear music.
Technical explanation
A neural network computes ŷ = f(W_n · ... · σ(W_2 · σ(W_1 · x))). Training minimizes a loss L(y, ŷ) — cross-entropy for classification, MSE for regression. Backpropagation applies the chain rule layer-by-layer: ∂L/∂W_i = ∂L/∂ŷ · ∂ŷ/∂z_i · ∂z_i/∂W_i. Gradient descent updates each weight: W ← W − η · ∂L/∂W. Modern variants (Adam) maintain per-parameter momentum (m) and variance (v) estimates: m ← β₁m + (1−β₁)g, v ← β₂v + (1−β₂)g², W ← W − η·m̂/√v̂. Batch normalization stabilizes activations between layers, preventing vanishing/exploding gradients. Dropout randomly zeros activations during training, acting as implicit ensemble regularization. The number of multiply-accumulate operations per forward pass is 2×(parameter count), so a 7B parameter model does ~14 billion FLOPs per token at inference.
Architecture
A deep learning training loop: DataLoader (batches x, y) → Forward pass (compute ŷ) → Loss function (scalar L) → loss.backward() (auto-differentiation fills .grad on every tensor) → optimizer.step() (updates weights) → optimizer.zero_grad() (clears accumulated gradients). The computation graph is built dynamically (PyTorch) or statically (JAX/XLA). Distributed training shards this loop: DDP replicates the model across GPUs and averages gradients via all-reduce; FSDP shards both parameters and optimizer state to fit 70B+ models.
Workflow
- Initialize weights -> 2. Run batch -> 3. Evaluate loss -> 4. Backpropagate error -> 5. Update weights -> 6. Repeat.
Example
Minimal training loop (PyTorch)
import torch, torch.nn as nn model = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10)) opt = torch.optim.Adam(model.parameters(), lr=1e-3) loss_fn = nn.CrossEntropyLoss()
for x, y in dataloader: # x: (B,784), y: (B,) logits = model(x) # forward pass loss = loss_fn(logits, y) # scalar loss opt.zero_grad() # clear stale gradients loss.backward() # fill .grad on all parameters opt.step() # W ← W - lr * grad
Gradient check: weight update magnitude
for name, p in model.named_parameters(): ratio = (lr * p.grad.std() / p.data.std()).item() print(f'{name}: update/weight ratio = {ratio:.4f}') # want ~1e-3
Real-world usage
Pre-training GPT-4 used ~25,000 A100 GPUs for months, consuming ~2×10²³ FLOPs. Each training step processes a batch of 2048 token sequences; the optimizer maintains dual fp32 copies of every weight (master weights + bf16 compute weights). The loss curve should drop steeply in the first 1% of steps; a flat curve signals a dead learning rate or corrupted batch. Fine-tuning a 7B model on domain data takes 4–8 hours on a single A100 with LoRA; the total trainable parameters shrink from 7B to ~4M rank-16 adapter weights.
Trade-offs
Larger batch sizes make gradient updates stable but require massive GPU memory.
Visual explanation
Loss surface as a 3-D landscape — valleys are minima, ridges are saddle points: High loss plateau ──[gradient points downhill]──> Valley (low loss) Learning rate too high: overshoots valley, bounces between walls Learning rate too low: crawls, may get trapped in local minimum Adam: adds momentum to roll past small bumps, and per-param scaling to handle sparse features
Backpropagation signal flow (Jay Alammar style): [Loss ∂L/∂ŷ] ──> [Output layer ∂L/∂W_n] ──> [Hidden ∂L/∂W_n-1] ──> ... ──> [Input ∂L/∂W_1] Vanishing gradient: signal shrinks with depth (sigmoid squashes ∂ to ≈0) ReLU fix: ∂ReLU/∂z = 1 if z>0, so gradient flows unchanged through positive activations
Advantages
- —
Scale-invariant (model quality increases with compute and parameters)
- —
End-to-end learning (no hand-engineered features needed)
Disadvantages
- —
Extremely computationally expensive
- —
Can overfit training data, failing on new samples
🎤 Interview questions
Explain backpropagation. How is the chain rule used to calculate gradients of deep layers?
📂 Subtopics
Gradient Descent Variants: SGD, Momentum and Adam
Plain gradient descent takes small steps opposite the gradient, but it's slow and jittery. Momentum smooths the path; Adam adapts the step size per weight — the reason Adam is the default optimizer in most modern training.
~14 min
Overfitting and Regularization: Dropout, Weight Decay and Early Stopping
A model that memorizes its training data instead of learning general patterns is overfit — great on training loss, bad on new data. Dropout, weight decay and early stopping are three standard ways to prevent it.
~13 min
Batch Normalization: Why Deep Networks Are Hard to Train and How It Helps
Deep networks suffer from 'internal covariate shift' — each layer's input distribution keeps shifting as earlier layers update. BatchNorm re-centers and re-scales activations at every layer, making training faster and more stable.
~12 min
Training Tips and Tricks: LR Scheduling, Gradient Clipping and Mixed Precision
Three practical techniques that make real training runs work: learning rate scheduling (change the step size over time), gradient clipping (cap runaway gradients), and mixed precision (train faster with lower-precision numbers).
~13 min