Stage 1 — Pre-training: Teaching the Basics of Language

A brand-new LLM begins life as a randomly initialized set of weights. Ask it 'What is an LLM?' at this stage and you'd get gibberish — it has seen zero data and has no notion of language at all.

Pre-training is the first and most expensive stage that fixes this. The model is trained on massive text corpora (a large fraction of the publicly available internet, books, code, and more) with a single, simple objective: predict the next token given everything before it. Repeating this next-token-prediction objective billions of times across trillions of tokens is what teaches the model grammar, factual knowledge, reasoning patterns, and even some coding ability — all as a side effect of getting good at 'what word comes next.'

The catch: a pre-trained model is not yet a chat assistant. If you prompt it with a question, it doesn't answer — it just continues the text in whatever direction is statistically most likely, which might mean generating more questions in the same style, or continuing an essay, rather than replying conversationally. This is exactly why pre-training alone isn't shipped as a product — it's the foundation that the next three stages (instruction fine-tuning, preference fine-tuning, reasoning fine-tuning) build a conversational, aligned assistant on top of.

Because pre-training corpora and compute are so large, this stage is distributed across many GPUs in parallel — the model's parameters, gradients, and the training data itself are partitioned across the cluster so it can process the scale of data required in a feasible amount of time.

# Conceptual sketch of the next-token-prediction training objective —
# real pre-training runs this at a trillion-token, multi-GPU scale, but
# the core loop is exactly this simple idea repeated at massive scale.
import torch
import torch.nn.functional as F

def pretraining_step(model, token_ids: torch.Tensor) -> torch.Tensor:
    # token_ids: (batch, seq_len) — a chunk of raw text, tokenized
    inputs = token_ids[:, :-1]   # everything except the last token
    targets = token_ids[:, 1:]   # everything shifted one to the right

    logits = model(inputs)  # (batch, seq_len-1, vocab_size)
    loss = F.cross_entropy(
        logits.reshape(-1, logits.size(-1)),
        targets.reshape(-1),
    )
    return loss  # backprop + optimizer step happens on this, repeated
                 # over trillions of tokens across many GPUs