What Is Reasoning in LLMs: Chain-of-Thought and Why Explicit Steps Help

An LLM fundamentally does one thing: predict the next token, given everything generated so far. When asked 'what's 47 times 23,' a model that jumps straight to an answer has to produce that answer using roughly the SAME fixed amount of computation it would spend generating any other single token — there's no built-in mechanism for 'thinking harder' about a hard question than an easy one, if the answer has to appear immediately as the very next token.

This is the structural problem 'reasoning' in LLMs addresses. Rather than asking for the answer directly, you nudge the model to first generate intermediate reasoning steps — 'first, multiply 47 by 20, then multiply 47 by 3, then add the results' — before the final answer. This matters for a genuinely mechanical reason, not just a stylistic one: each of those intermediate steps is itself generated with the model's normal per-token computation budget, so producing 10 steps of reasoning gives the model roughly 10x the total computation to work with before committing to a final answer, compared to answering immediately. And crucially, each step becomes part of the CONTEXT for the next one — the model isn't just 'thinking longer' in some vague sense, it's literally conditioning each subsequent prediction on its own prior reasoning, the same way it conditions on any other text in its context window.

This pattern is widely known as Chain-of-Thought (CoT) prompting or reasoning — the simplest and most widely used reasoning technique, covered in book-grounded detail (alongside two related techniques) in this topic's own 'Techniques' subtopic. The core mechanism worth internalizing here, before the specific techniques: explicit reasoning steps aren't a magic trick that makes the model smarter in some abstract sense — they're a way of trading more GENERATED TOKENS (and thus more total computation) for a better chance at a correct final answer, on tasks where the answer genuinely benefits from multi-step derivation rather than pattern recall.

This framing also previews the 'reasoning models' subtopic later in this topic: if explicit reasoning steps help because they buy more computation before the final answer, then a natural next idea is training a model SPECIFICALLY to generate long, effective reasoning chains as its default behavior, rather than relying on a human's prompt to request them — which is exactly what dedicated reasoning models do differently from a base LLM prompted with 'think step by step.'

# Illustrating the mechanical difference CoT makes: direct-answer
# generation gets ONE token's worth of 'thinking' before committing,
# while CoT generation gets N tokens' worth spread across visible steps.

def direct_answer(question: str) -> str:
    """Simulates jumping straight to an answer -- one generation step,
    one token's worth of computation budget before committing."""
    return "[answer generated with 1 step's worth of computation]"

def chain_of_thought_answer(question: str, reasoning_steps: list[str]) -> dict:
    """Simulates CoT: each step is generated conditioned on all PRIOR
    steps, giving the model many steps' worth of computation, and each
    step becomes context for the next."""
    accumulated_context = question
    trace = []
    for i, step in enumerate(reasoning_steps):
        # Each step is conditioned on everything generated so far --
        # exactly like normal next-token prediction, just applied to
        # intermediate reasoning rather than jumping to the final answer
        accumulated_context += f"\nStep {i+1}: {step}"
        trace.append(step)
    final_answer = "[answer generated with N steps' worth of computation]"
    return {"trace": trace, "final_answer": final_answer, "total_steps": len(reasoning_steps) + 1}

question = "What is 47 * 23?"
print("Direct:", direct_answer(question))

cot_result = chain_of_thought_answer(question, [
    "47 * 20 = 940",
    "47 * 3 = 141",
    "940 + 141 = 1081",
])
print("Chain-of-Thought:", cot_result)
print(f"CoT used {cot_result['total_steps']}x the generation steps of the direct answer")