Gradient Descent Variants: SGD, Momentum and Adam
~14 min read
Plain gradient descent takes small steps opposite the gradient, but it's slow and jittery. Momentum smooths the path; Adam adapts the step size per weight — the reason Adam is the default optimizer in most modern training.
Gradient Descent Variants: SGD, Momentum and Adam is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •Gradient descent repeatedly steps weights opposite their gradient — like a hiker walking downhill on the loss landscape
- •Plain SGD zig-zags and converges slowly in 'ravine' shaped landscapes because it only reacts to the current gradient
- •Momentum accumulates a running average of past gradients (like a rolling ball building speed), smoothing zig-zags and speeding up consistent directions
- •Adam tracks a per-weight running average of both the gradient (direction) and its square (magnitude), giving each weight its own adaptive step size
- •Adam's combination of momentum-style smoothing plus per-weight adaptive steps is why it's the default optimizer for most deep learning, including LLM training