LLM Reasoning & Test-Time Compute
How modern LLMs perform multi-step reasoning and why test-time compute changes the capability ceiling
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Understand why single-pass generation fails on complex tasks
- •Explain Chain-of-Thought and its variants
- •Describe how o1/o3-style reasoning loops work
- •Know when to use reasoning models vs standard models in production
- •Calculate cost vs quality tradeoff for reasoning models
What is it?
LLM reasoning refers to techniques — from simple Chain-of-Thought prompting to dedicated reasoning models like OpenAI's o1/o3 and DeepSeek's R1 — that give a model more effective compute at inference time before it commits to a final answer, dramatically improving reliability on complex multi-step problems like math, logic, and code. The core idea: generating intermediate reasoning steps lets the model condition each new prediction on its own prior reasoning, trading extra generated tokens for a better shot at correctness — the foundation of 'test-time compute' as a new axis for scaling capability, alongside training-time scale.
Why it exists
Standard LLMs generate one token at a time in a single pass. Complex tasks require intermediate steps — verification, backtracking, self-correction. Test-time compute scaling lets models "think longer" to get better answers.
Problem it solves
LLMs confidently giving wrong answers on math, logic puzzles, and multi-step planning tasks where a single forward pass is insufficient.
Intuition
Think of it like the difference between a student blurting out an answer vs one who writes down their work. The student who writes intermediate steps catches their own mistakes.
Analogy
A single-pass LLM is like answering a crossword puzzle by filling every square in one sweep. A reasoning model is like solving it iteratively — filling what you know, using those answers to unlock harder clues, erasing mistakes.
Technical explanation
Chain-of-Thought (CoT) prompting instructs the model to output reasoning steps before the final answer. Self-Consistency samples multiple CoT paths and takes the majority answer. Process Reward Models (PRMs) score intermediate reasoning steps rather than just final answers. o1/o3-style models use a learned internal reasoning loop — they generate a hidden "thinking" trace (using a special reasoning token budget) before producing the visible response. GRPO (Group Relative Policy Optimization) trains models to reason better by reinforcing correct reasoning paths. Monte Carlo Tree Search (MCTS) can be applied to LLM decoding to explore multiple reasoning branches.
Architecture
Standard: prompt → single forward pass → answer. CoT: prompt + "think step by step" → reasoning trace → answer. o1-style: prompt → [hidden thinking tokens, N×forward passes] → answer. PRM-guided: prompt → candidate answers via beam search → PRM scores each → best answer returned.
Workflow
- Identify if task needs reasoning (math, logic, multi-step planning).
- Try CoT prompting first (zero-cost).
- If unreliable, use self-consistency (sample 5-10, majority vote).
- If still failing, use a reasoning model (o1, o3, claude-3-7-sonnet).
- Monitor cost — reasoning models cost 5-20x more per query.
Example
Task: "Is 17 x 23 = 391?" Standard LLM: "Yes" (often wrong). CoT LLM: "17 x 20 = 340, 17 x 3 = 51, 340 + 51 = 391. Yes." Reasoning model: runs internal verification loop, higher confidence.
Real-world usage
Coding assistants (complex bug fixing), math tutors, legal document analysis, multi-step data analysis pipelines. NOT worth it for: simple Q&A, summarization, classification — too expensive.
Trade-offs
Cost vs accuracy: reasoning models are only worth it when task complexity justifies the price premium. Latency vs reliability: reasoning adds 2-10 seconds of thinking time.
Visual explanation
Standard: [prompt] → [single forward pass] → [answer] CoT: [prompt + "think step by step"] → [reasoning trace] → [answer] o1-style: [prompt] → [hidden thinking N×passes] → [answer]
Advantages
- —
Dramatically better on complex multi-step tasks
- —
Self-correction catches errors before output
- —
Test-time compute scales independently of model size
Disadvantages
- —
5-20x higher cost per query vs standard models
- —
Slower latency (seconds vs milliseconds)
- —
Overkill for simple tasks
- —
Hidden reasoning trace not always inspectable
Common mistakes
- —
Using reasoning models for all tasks regardless of complexity (huge cost waste)
- —
Not setting a reasoning budget/token limit
- —
Ignoring that CoT prompting on a standard model often gets 80% of the benefit at 1% of the cost
🎤 Interview questions
How does o1 differ from GPT-4 architecturally?
What is test-time compute scaling?
When would you NOT use a reasoning model in production?
Explain the difference between outcome reward models and process reward models.
What is GRPO and how does it improve reasoning?
📂 Subtopics
What Is Reasoning in LLMs: Chain-of-Thought and Why Explicit Steps Help
LLMs generate one token at a time with a fixed amount of computation per token — 'reasoning' is what happens when a model is nudged to spend more of that computation on visible intermediate steps before committing to an answer.
~12 min
Reasoning vs Memorization: When LLMs Actually Reason vs Pattern-Match
Much of what LLMs do is closer to sophisticated pattern-matching than genuine derivation — and telling the two apart matters for knowing when to trust an answer.
~12 min
Reasoning Techniques: Chain of Thought, Self-Consistency, and Tree of Thoughts
The book's 3 named prompting techniques for reasoning — CoT (reason step by step), Self-Consistency (vote across multiple reasoning paths), and Tree of Thoughts (search across branching reasoning paths) — each building on the last.
~14 min
Reasoning Models: o1/o3/R1-Style Models and Test-Time Compute
A newer class of models is trained to generate long internal reasoning by default, rather than relying on a human's prompt to request it — trading more inference-time compute for better performance on hard reasoning tasks.
~13 min