What is an LLM & How It works
Learn the core neural architectures, conditional probability, tokens, and dense vs MoE configurations.
Token representations input
Text sequences are projected into high-dimensional vector spaces representing query indices.
Transformer Architecture
This diagram shows the core data flow inside a Transformer model — the architecture behind all modern LLMs. Input tokens are first converted to embeddings, then combined with positional encodings so the model knows token order. The Multi-Head Attention mechanism lets every token attend to every other token simultaneously (the key innovation over RNNs). The Feed Forward Network then processes each position independently. Layer Norm is applied after each sub-layer to stabilise training. Understanding this flow explains why LLMs can handle long contexts, why attention is the bottleneck at inference time, and why positional encoding matters.
Transformer Architecture
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Grasp the conditional probability model of next-token prediction P(next_token | previous_tokens)
- •Understand tokenization, vocabulary maps, and positional encodings
- •Explain the core difference between standard dense Transformers and Mixture of Experts (MoE)
What is it?
A Large Language Model (LLM) is a neural network with billions of parameters trained on massive text corpora to predict the next token given a sequence of previous tokens. The core architecture is the Transformer — a stack of attention layers that learn relationships between tokens at any distance. LLMs like GPT-4, Claude, and Gemini are not retrieving stored answers — they compute probability distributions over vocabularies, token by token, generating output one piece at a time.
Why it exists
Before LLMs, NLP required task-specific models — one for translation, one for summarization, one for classification. Each needed hand-crafted features and separate training data. LLMs emerged from the insight that a model trained to predict text at scale implicitly learns grammar, facts, reasoning, and coding — making a single model useful across all NLP tasks without task-specific fine-tuning.
Problem it solves
LLMs solve the fragmentation problem: before them, building an AI feature required a pipeline of specialized models (intent classifier → entity extractor → response generator). Now one model handles the full pipeline with just a prompt. They also solve the labeled data bottleneck — LLMs learn from raw internet text, not expensive human-labeled datasets.
Intuition
Think of an LLM as a very sophisticated autocomplete — but autocomplete that has read essentially all human text ever written. When you say "The capital of France is", it completes "Paris" not because it looked it up, but because that pattern appeared so many times in training that "Paris" is overwhelmingly probable. The "intelligence" emerges from learning statistical patterns across billions of documents.
If you come from Java/Spring Boot: an LLM is like a giant dependency injection container — instead of wiring beans, it wires concepts. Pretraining builds the framework; prompting is writing your application on top of it. You do not modify the framework — you call its API.
If you come from React/Frontend: think of an LLM as a universal component — you pass it props (your prompt) and it renders output (the completion). Fine-tuning is like creating a theme — same base component, different behavior. The model has no persistent state between calls, just like a pure functional component.
Analogy
An LLM is like a chef who has read every recipe book ever written. Ask them to cook anything and they can — not because they memorized exact recipes, but because they internalized patterns: acid balances fat, heat denatures protein, sugar caramelizes at 160°C. They generate new dishes by combining learned patterns, not by retrieving stored recipes. Similarly, an LLM generates new text by combining statistical patterns learned during training.
Technical explanation
Transformer architecture: input tokens are embedded into vectors (dim 512–4096), then passed through N layers of Multi-Head Self-Attention (MHSA) + Feed-Forward Networks (FFN). MHSA computes: Attention(Q,K,V) = softmax(QKᵀ / sqrt(d_k)) × V. This operation is O(n²) in sequence length — the fundamental scaling bottleneck.
Training: next-token prediction on a corpus (15 trillion tokens for Llama 3). Loss = cross-entropy between predicted and actual next token. Gradient descent updates all ~70B parameters per step. Training a frontier model costs $50–100M in compute.
Inference: autoregressive — one token per forward pass. A 4K-token response = 4K forward passes. Each pass through a 70B model requires reading ~140GB of weights from GPU memory — inference latency is dominated by memory bandwidth, not raw compute.
Key architectural details: modern LLMs use RMSNorm instead of LayerNorm, RoPE (Rotary Position Embedding) instead of absolute positional encodings, SwiGLU activation in FFN layers. These changes improve training stability and enable longer context windows.
Mixture of Experts (MoE): instead of one dense FFN per layer, MoE uses N expert FFNs and routes each token to K of them (K=2 of N=8 is common). GPT-4 is believed to be a ~1.8T parameter MoE — only ~200B parameters activate per token, giving frontier capability at dense-200B inference cost.
Architecture
LLM Forward Pass (decoder-only, e.g. GPT/Llama):
Input text → Tokenizer (BPE) → token IDs → Embedding lookup (vocab_size × d_model matrix) → Add RoPE positional encoding → [Layer 1 to N]: RMSNorm → Multi-Head Self-Attention (Q, K, V projections) → Residual add → RMSNorm → Feed-Forward Network (SwiGLU, 4× expansion) → Residual add → Final RMSNorm → LM Head (d_model × vocab_size) → Softmax → token probabilities → Sample next token (temperature + top-p)
Llama 3 70B dimensions: d_model=8192, n_heads=64, n_layers=80, FFN_hidden=28672, context=128K
Workflow
- Tokenize: "Hello world" → [15496, 995] via BPE tokenizer
- Embed: token IDs → dense vectors via embedding matrix lookup
- Positional encoding: RoPE encodes position into Q/K matrices directly
- N transformer layers: each applies MHSA (with KV cache on previous tokens) then FFN
- LM head: project final hidden state to logits over vocabulary (shape: [vocab_size])
- Sample: apply temperature scaling, top-p filtering, then multinomial sample
- Append generated token to sequence, repeat from step 3 until EOS or max_tokens
- KV cache optimization: K/V matrices for all previous tokens are cached — only the new token needs fresh computation, reducing per-step cost by ~60–80%
Example
Python + Anthropic SDK:
import anthropic client = anthropic.Anthropic()
response = client.messages.create( model="claude-opus-4-5", max_tokens=1024, messages=[{"role": "user", "content": "Explain attention in 2 sentences"}] ) print(response.content[0].text)
Cost: 1024 output tokens × $15/1M = $0.015 per call
Streaming version for real-time output:
with client.messages.stream(model="claude-opus-4-5", max_tokens=512, messages=[{"role": "user", "content": "What is RAG?"}]) as stream: for text in stream.text_stream: print(text, end="", flush=True)
Real-world usage
OpenAI GPT-4 powers ChatGPT (200M+ users), GitHub Copilot (1M+ developers), Microsoft 365 Copilot. Uses MoE architecture, 128K context window. GPT-4o adds native multimodal (vision + audio).
Anthropic Claude 3 is used by Notion AI, Slack AI, Amazon Bedrock. Constitutional AI training method. Claude 3 Opus has 200K context — large enough to analyze full codebases in a single call.
Google Gemini 1.5 Pro has a 1M-token context window. Powers Google Search AI Overviews, Google Docs, Gmail Smart Compose. Native multimodal from the ground up — handles text, images, video, audio.
Trade-offs
Cost vs capability: GPT-4o costs ~$5/1M input tokens; GPT-4o-mini costs $0.15/1M — 33× cheaper with 70–80% of the capability for most tasks. Match model size to task complexity.
Context length vs cost: doubling context roughly doubles KV cache memory and increases compute. 128K context is ~32× more expensive per token than 4K context. Only use long context when you actually need it.
Latency vs quality: reasoning models (o1, o3) spend 10–60s "thinking" before responding — dramatically better on math/code but unusable for real-time chat. Standard models respond in <1s.
Visual explanation
Dense Transformer vs. Mixture of Experts (MoE): ┌──────────────────────────────┐ ┌──────────────────────────────────┐ │ DENSE TRANSFORMER │ │ SPARSE MIXTURE OF EXPERTS │ │ [Input Tokens] │ │ [Input Tokens] │ │ ↓ │ │ ↓ │ │ [Self-Attention Layer] │ │ [Self-Attention Layer] │ │ ↓ │ │ ↓ │ │ [Feed-Forward Network (All)]│ │ [Router / Gating Network] │ │ ↓ │ │ / | \ │ │ [Output Token Prediction] │ │ [Expert 1] [Expert 2] [Expert 3] │ └──────────────────────────────┘ └──────────────────────────────────┘
Advantages
- —
General-purpose natural language reasoning
- —
Highly contextual multi-turn comprehension
- —
Single model handles translation, summarization, QA, code generation without task-specific retraining
Disadvantages
- —
Heavy computational footprint for training and serving
- —
Vulnerable to hallucinations and predictability biases
- —
Training data cutoff means knowledge is frozen at a point in time
Common mistakes
- —
Using max_tokens too high for simple tasks — if you want a yes/no classification but set max_tokens=4096, you waste inference budget and slow responses. Set max_tokens to 10-20 for classification tasks.
- —
Ignoring token counting when building prompts — 128K context sounds huge but 500 source files easily exceed it. Always measure token usage with a tokenizer library before assuming content fits in context.
- —
Treating temperature=0 as fully deterministic — floating point operations across different hardware can produce slightly different results even at temp=0. For true reproducibility, also set a fixed seed where the API supports it.
- —
Not using prompt caching for repeated system prompts — if your system prompt is 2000 tokens and you make 100K calls/day, that is 200M cached tokens/day. Anthropic and OpenAI both offer 80-90% cost reduction on cached prefixes.
- —
Using a frontier model for every task — Llama 3.1 8B handles classification, extraction, and simple generation at 1/100th the cost of GPT-4. Benchmark your specific task on smaller models before defaulting to GPT-4.
🎤 Interview questions
Explain the difference between Dense Transformers and Sparse MoE models. How does routing affect inference throughput?
📂 Subtopics
Tokenization: How LLMs Read Text
Before an LLM can process any text, it must convert it into tokens — integer IDs from a fixed vocabulary. Understanding tokenization explains why LLMs behave differently on different languages and why costs vary.
~15 min
Self-Attention: How Transformers Understand Context
Self-attention is the mechanism that lets each token look at every other token when being processed. It is why transformers understand long-range dependencies that RNNs struggled with.
~25 min
Sampling & Temperature: Controlling Randomness
How an LLM picks its next token is controlled by sampling parameters. Temperature, top-p, and top-k determine whether output is deterministic or creative.
~10 min