The Forward Pass: How Input Flows Through Layers via Matrix Multiplication

The previous subtopic described one neuron's computation with a for-loop over weights. That's correct but slow and doesn't reveal WHY neural networks run so fast on modern hardware. The forward pass — the process of pushing an input all the way through the network to produce an output — is really just matrix multiplication, done layer by layer.

Here's the reframing: instead of thinking of a layer as 'a bunch of separate neurons,' think of it as ONE weight matrix W (rows = neurons in this layer, columns = inputs from the previous layer) and one bias vector b. The whole layer's output, for every neuron at once, is a single matrix operation: output = activation(W @ x + b), where @ is matrix-vector multiplication and activation is applied elementwise to every entry of the result. This is exactly equivalent to running every neuron's weighted-sum-plus-bias individually — it's just expressed as one matrix operation instead of many small loops, which computers (especially GPUs, purpose-built for exactly this kind of bulk arithmetic) execute far faster than an equivalent Python for-loop.

The forward pass chains this across every layer: x1 = activation(W1 @ x0 + b1), x2 = activation(W2 @ x1 + b2), and so on, where x0 is your raw input and the final xN is the network's output. Each layer's output becomes the next layer's input, which is why this direction is called 'forward' — information flows strictly from input toward output, layer by layer, with no loops back.

For a whole BATCH of inputs at once (not just one example), you stack the individual input vectors into rows of a matrix X, and the same formula (with slightly adjusted shapes) computes every example's forward pass simultaneously in one matrix multiplication — this batching is a huge part of why training on GPUs, which excel at large matrix multiplications, is dramatically faster than looping over examples one at a time. Every framework you'll use in practice (PyTorch, TensorFlow) is, underneath the convenience API, doing exactly this chain of matrix multiplications and elementwise activations.

# The same tiny MLP as the previous subtopic, rewritten as matrix
# multiplication instead of per-neuron loops -- the real implementation.

def matvec(W: list[list[float]], x: list[float]) -> list[float]:
    """Matrix-vector product: one row of W per output neuron."""
    return [sum(w_ij * x_j for w_ij, x_j in zip(row, x)) for row in W]

def add(a: list[float], b: list[float]) -> list[float]:
    return [ai + bi for ai, bi in zip(a, b)]

def relu_vec(z: list[float]) -> list[float]:
    return [max(0.0, zi) for zi in z]

def forward_pass(x0: list[float], layers: list[tuple]) -> list[float]:
    """Chain matrix multiplications through every layer:
    x_{i+1} = activation(W_i @ x_i + b_i)"""
    x = x0
    for W, b, activation in layers:
        z = add(matvec(W, x), b)
        x = activation(z)
    return x

# Same network as before: 2 inputs -> 3 hidden neurons -> 1 output
W_hidden = [[0.5, -0.3], [-0.2, 0.8], [0.1, 0.1]]
b_hidden = [0.1, 0.0, -0.2]
W_out = [[0.4, 0.4, 0.4]]
b_out = [0.0]

layers = [(W_hidden, b_hidden, relu_vec), (W_out, b_out, relu_vec)]
result = forward_pass([1.0, 0.5], layers)
print(f"forward pass output: {result}")
# With real libraries: torch.nn.Linear(2, 3) does W@x + b in one call