From a Single Neuron to a Multi-Layer Perceptron

Think of a single artificial neuron like a tiny voting booth. It receives several numeric inputs (x1, x2, x3, ...), and each input gets a 'vote weight' (w1, w2, w3, ...) reflecting how important that input is. The neuron multiplies each input by its weight, adds them all up, adds one more adjustable number called a bias (a built-in lean toward yes or no, independent of the inputs), and that sum is the neuron's raw output: z = (w1x1 + w2x2 + ... ) + bias.

If we stopped there, stacking neurons would be pointless — a sum of sums of sums is still just one big sum, mathematically no more powerful than a single neuron (this is a real limitation of the very first neural network, the 1958 Perceptron, which could only draw a straight line to separate two classes). The fix is the activation function: a nonlinear function applied to z before passing it on. Three you'll see constantly: Sigmoid squashes any number into the range (0, 1), useful for outputs you want to read as a probability. Tanh squashes into (-1, 1), like sigmoid but centered at zero. ReLU (Rectified Linear Unit) is the simplest and most-used today — it's just max(0, z): if z is positive, pass it through unchanged; if negative, output zero. Because ReLU is so cheap to compute and avoids some training problems the others have (covered in a later subtopic), it's the default choice inside most modern networks.

A single neuron with an activation function can only draw one curve of separation. The real power comes from stacking many neurons into LAYERS, and stacking many layers into a network — this is a Multi-Layer Perceptron (MLP). The first layer (input layer) receives your raw data. One or more hidden layers each take the PREVIOUS layer's outputs as their inputs, apply their own weights/biases/activation, and pass the result forward. The final layer (output layer) produces the answer — a single number, a probability, or a whole distribution (often via softmax, from the probability-basics unit). Each hidden layer lets the network combine simpler patterns from the layer before it into more complex ones, which is the whole reason 'deep' (many-layered) networks can learn far richer patterns than a single neuron ever could.

import math

def relu(z: float) -> float:
    return max(0.0, z)

def sigmoid(z: float) -> float:
    return 1 / (1 + math.exp(-z))

class Neuron:
    """A single neuron: weighted sum of inputs + bias, then activation."""
    def __init__(self, weights: list[float], bias: float, activation=relu):
        self.weights = weights
        self.bias = bias
        self.activation = activation

    def forward(self, inputs: list[float]) -> float:
        z = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        return self.activation(z)

class Layer:
    """A layer is just several neurons, each seeing the same inputs."""
    def __init__(self, neurons: list[Neuron]):
        self.neurons = neurons

    def forward(self, inputs: list[float]) -> list[float]:
        return [n.forward(inputs) for n in self.neurons]

# A tiny MLP: 2 inputs -> hidden layer of 3 neurons -> output layer of 1
hidden = Layer([Neuron([0.5, -0.3], 0.1), Neuron([-0.2, 0.8], 0.0),
                Neuron([0.1, 0.1], -0.2)])
output = Layer([Neuron([0.4, 0.4, 0.4], 0.0, activation=sigmoid)])

x = [1.0, 0.5]
h = hidden.forward(x)
y = output.forward(h)
print(f"hidden layer output: {h}")
print(f"final output (probability-like): {y}")