Introduction to Neural Networks
Learn neurons, connection weights, biases, layers, activation functions (ReLU), and how signals propagate forward.
Input Layer Initialization
Raw features (such as vectors of word embeddings) are fed into the input neurons.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Grasp how a artificial neuron multiplies inputs by weights and adds a bias
- •Explain activation functions like ReLU and Sigmoid
- •Track feed-forward signal flow through multiple hidden layers
What is it?
A neural network is a function approximator made of connected layers of mathematical operations. Each layer applies a linear transformation (matrix multiplication) followed by a non-linear activation function. By stacking dozens to hundreds of such layers with billions of parameters, neural networks can approximate virtually any function — including the mapping from "text prompt" to "next word probability distribution" that powers LLMs.
Why it exists
Before neural networks, engineers had to manually define features (e.g., "is this email spam?" required writing rules about word frequencies, sender reputation, etc.). Neural networks learn these features automatically from examples. This scales: more data + more parameters = better features = better performance, without manual feature engineering.
Problem it solves
Neural networks solve tasks where the mapping from input to output is too complex to write rules for: image recognition, speech synthesis, language translation, and predicting the next token in a sequence. The Transformer neural network architecture is the foundation of every modern LLM.
Intuition
Each neuron is a tiny decision-maker. It receives numbers, multiplies each by its learned importance (weight), sums them up, adds a bias, and passes the result through an activation function that decides how loudly to "fire." Stacking millions of these creates a network that can learn any pattern from data.
If you come from Java/Spring Boot: a neural network layer is like a service method that takes an input array, applies a linear transformation (like a matrix of configuration values), and produces a transformed output array. The "weights" are like @Autowired configuration values — but instead of being set by a developer, they are learned automatically from training data by gradient descent.
If you come from React/Frontend: a neural network is like a tree of React components, each transforming its input before passing to the next. The weights are like component props — but they are automatically tuned by backpropagation rather than manually set. Forward pass = rendering the component tree. Backpropagation = automatically adjusting all the props to minimize a loss (error) metric.
Analogy
A neural network is like a committee of judges scoring a gymnastics routine. Each judge (neuron) watches specific aspects (features) of the performance. They assign importance weights to what they see, sum their scores, and pass their verdict to the next panel. After thousands of routines, the judges have learned which features truly matter for a great score — without anyone telling them what to look for.
Technical explanation
Single neuron computation: y = f(Σ wᵢxᵢ + b) = f(w₁x₁ + w₂x₂ + ... + wₙxₙ + b) where w = weight vector, x = input vector, b = bias scalar, f = activation function
Matrix form for a full layer: H = f(X @ W + b) where X ∈ ℝ^(batch×input_dim), W ∈ ℝ^(input_dim×output_dim), b ∈ ℝ^output_dim
Activation functions: ReLU: f(x) = max(0, x) — most common, kills negative values, enables sparse activations GELU: f(x) = x × Φ(x) — smoother than ReLU, used in GPT/BERT SiLU/Swish: f(x) = x × sigmoid(x) — used in LLaMA (as part of SwiGLU) Softmax: converts logits to probabilities: eˣⁱ / Σeˣʲ — output layer of LLMs
Why non-linearity matters: without activation functions, stacking N linear layers is mathematically identical to one linear layer (W_total = W_n @ W_{n-1} @ ... @ W_1). Non-linearities allow the network to approximate non-linear functions.
Transformer connection: LLMs are neural networks with ~80 layers (Llama 3 70B) where each layer has Multi-Head Self-Attention + Feed-Forward Network. The FFN is exactly this: linear layer → SwiGLU activation → linear layer.
Architecture
Multi-layer perceptron (MLP) example: Input layer: [x₁, x₂, x₃] (3 features) Hidden layer 1: 128 neurons, ReLU activation Hidden layer 2: 64 neurons, ReLU activation Output layer: 10 neurons, Softmax (10-class classification)
Data flow: x → W₁·x+b₁ → ReLU → W₂·h₁+b₂ → ReLU → W₃·h₂+b₃ → Softmax → probabilities
Workflow
- Define architecture: number of layers, neurons per layer, activation functions
- Initialize weights: random (Xavier/Kaiming initialization to avoid vanishing/exploding gradients)
- Forward pass: compute predictions by propagating input through all layers
- Compute loss: compare predictions to true labels (cross-entropy for classification)
- Backward pass: compute gradients via chain rule (backpropagation)
- Update weights: optimizer (SGD, Adam) adjusts weights to reduce loss
- Repeat 3-6 for thousands of batches until convergence
Example
import numpy as np
Single layer forward pass
def relu(x): return np.maximum(0, x)
Layer: 2 inputs → 3 hidden neurons → 1 output
W1 = np.random.randn(2, 3) * 0.1 # Xavier init b1 = np.zeros(3) W2 = np.random.randn(3, 1) * 0.1 b2 = np.zeros(1)
Forward pass
x = np.array([0.5, 0.8]) h1 = relu(x @ W1 + b1) # hidden layer, shape (3,) out = h1 @ W2 + b2 # output, shape (1,) print(f"Prediction: {out[0]:.4f}")
In LLMs, W1 and W2 have dimensions like (8192, 28672) — 237M params per layer
Real-world usage
Every LLM is a stack of neural network layers. Llama 3 70B has 80 transformer layers, each containing: Multi-Head Self-Attention (the neural network that computes token relationships) + Feed-Forward Network (two linear layers with SwiGLU activation). Understanding single neurons → multi-layer perceptrons → transformers is the full conceptual ladder.
Trade-offs
Deeper networks (more layers) can represent more complex functions but are harder to train (vanishing/exploding gradients). Modern solutions: residual connections (ResNet/Transformer), layer normalization, and careful initialization. Wider networks (more neurons per layer) also increase capacity but cost more compute.
Visual explanation
Neuron architecture: Inputs [x1, x2] ──(Multiply by Weights w1, w2)──> Sum (w1x1 + w2x2) + Bias ──(Activation Function)──> Output
Advantages
- —
Can model highly non-linear patterns
- —
Improves automatically with more data
Disadvantages
- —
Acts as a black box (hard to explain why a decision was made)
- —
Requires massive training datasets
Common mistakes
- —
Forgetting to add non-linear activation functions between layers. Without activations, stacking 100 linear layers is mathematically equivalent to one linear layer. The entire point of depth is the non-linearity — every layer must have an activation function.
- —
Initializing all weights to the same value (e.g., zeros). If all weights are equal, all neurons in a layer receive identical gradients and learn the same thing — the layer effectively has one neuron, not N. Use random initialization (Xavier/He init) to break symmetry.
- —
Confusing model architecture with model weights. The architecture (number of layers, neurons) is fixed at design time. The weights are learned during training. Two models with the same architecture but different training data have completely different weights and behaviors.
- —
Not understanding that LLM attention layers ARE neural network layers. Self-attention is just matrix multiplications (Q, K, V projections) followed by softmax (an activation function) — it IS a neural network operation, just with a specific structure designed for sequence processing.
- —
Thinking bigger is always better. A 1B parameter model trained well on quality data outperforms a 7B model trained on noisy data. And a 7B model with proper fine-tuning outperforms GPT-4 for narrow tasks. Architecture and training matter more than raw size.
🎤 Interview questions
Walk through the forward pass equations of a single fully-connected neural layer. How are shapes matched?
📂 Subtopics
From a Single Neuron to a Multi-Layer Perceptron
A neuron just weighs its inputs, adds them up, and squashes the result through an activation function. Stack many neurons into layers and you have a Multi-Layer Perceptron — the basic shape of almost every neural network.
~13 min
The Forward Pass: How Input Flows Through Layers via Matrix Multiplication
The forward pass is just repeated matrix multiplication: each layer's neurons all compute their weighted sums at once as a single matrix-vector product, which is why GPUs (built for matrix math) are so good at running neural networks.
~12 min
Loss Functions: MSE, Cross-Entropy, and Why We Minimize Loss
A loss function is a single number measuring how wrong the network's output was. Training is nothing more than adjusting weights to make that number smaller — MSE for numeric predictions, cross-entropy for classification.
~12 min
Backpropagation: The Chain Rule and How Gradients Flow Backward
Backpropagation is just the chain rule from calculus, applied layer by layer in reverse — it tells every weight in the network exactly how much it contributed to the final error, so it knows which way to adjust.
~15 min