Stage 1 — Pre-training: Teaching the Basics of Language
~15 min read
Before any fine-tuning, an LLM starts as a randomly initialized model that knows nothing. Pre-training teaches it grammar, world facts, and next-token prediction by training on massive text corpora — but leaves it unable to hold a conversation.
Stage 1 — Pre-training: Teaching the Basics of Language is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •A randomly initialized LLM knows nothing — it produces gibberish until pre-training happens
- •Pre-training's objective is simple: predict the next token, repeated across massive text corpora
- •This single objective is enough to teach grammar, world facts, and reasoning patterns as a side effect
- •A pre-trained model completes text rather than conversing — it isn't yet an assistant
- •Pre-training is distributed across many GPUs because of the sheer scale of data and compute involved