Backpropagation: The Chain Rule and How Gradients Flow Backward

The forward pass computes a prediction; the loss function scores how wrong that prediction was. But knowing you're wrong doesn't tell you HOW to fix it — with potentially millions of weights scattered across many layers, which ones should change, and in which direction? Backpropagation is the algorithm that answers this, and despite its intimidating reputation, it's built entirely from one calculus idea you already sort of know: the chain rule.

The chain rule says: if changing A changes B, and changing B changes C, then the effect of A on C is the effect of A on B, MULTIPLIED by the effect of B on C. In a neural network, every weight affects the final loss through a long chain: a weight in an early layer changes that layer's output, which changes the next layer's output, which changes the one after that, all the way to the final prediction, which determines the loss. The chain rule lets you compute 'how much does THIS weight, way back in layer 2, affect the FINAL loss' by multiplying together a chain of much simpler 'how does this affect the next thing' derivatives, one layer at a time.

The word 'back' in backpropagation describes the DIRECTION you compute this in: you start at the loss (the very end) and work backward, layer by layer, toward the input. First you compute how much the loss changes with respect to the OUTPUT layer's values (easy — you have the loss formula directly). Then, using the chain rule, you use that to compute how much the loss changes with respect to the SECOND-TO-LAST layer's values. Then the layer before that. And so on, backward through the entire network. At each layer, you also compute how much the loss changes with respect to that layer's WEIGHTS specifically — this is the actual gradient you use to update that layer's weights. Each layer only needs the gradient signal handed to it from the layer AFTER it, plus its own local computation — it never needs to know about layers further away, which is what makes this algorithm efficient and modular.

Once you have the gradient for every weight (the direction that would make loss WORSE), you update each weight by taking a small step in the OPPOSITE direction — new_weight = old_weight - learning_rate * gradient. That single update rule is gradient descent, and it's the topic of the very next unit (deep-learning). Backpropagation is 'how do we get the gradients'; gradient descent is 'what do we do with them once we have them' — together they're the entire training loop for every neural network.

# A tiny worked example: one weight, one input, MSE loss --
# manually applying the chain rule to compute a gradient.

def forward(w: float, x: float) -> float:
    return w * x               # a trivially simple 'network': y = w*x

def loss_fn(y_pred: float, y_true: float) -> float:
    return (y_pred - y_true) ** 2    # MSE for a single example

def backward(w: float, x: float, y_true: float) -> float:
    """Chain rule: d(loss)/dw = d(loss)/d(y_pred) * d(y_pred)/dw
         d(loss)/d(y_pred) = 2 * (y_pred - y_true)     [derivative of (y_pred-y_true)^2]
         d(y_pred)/dw      = x                        [derivative of w*x w.r.t. w]
    """
    y_pred = forward(w, x)
    d_loss_d_ypred = 2 * (y_pred - y_true)
    d_ypred_d_w = x
    return d_loss_d_ypred * d_ypred_d_w   # the chain rule, multiplying two local derivatives

w, x, y_true = 0.5, 2.0, 3.0
y_pred = forward(w, x)
loss = loss_fn(y_pred, y_true)
gradient = backward(w, x, y_true)
print(f"prediction={y_pred}, loss={loss}, gradient dL/dw={gradient}")

learning_rate = 0.05
w_new = w - learning_rate * gradient   # step OPPOSITE the gradient (next unit: gradient descent)
print(f"updated weight: {w_new:.3f} (old loss={loss:.3f}, "
      f"new loss={loss_fn(forward(w_new, x), y_true):.3f})")