Reasoning Models: o1/o3/R1-Style Models and Test-Time Compute

The previous subtopic's three techniques (CoT, Self-Consistency, ToT) are all things a USER does — crafting a prompt to coax better reasoning out of a base model. This subtopic covers a different approach: training a model SPECIFICALLY to reason well by default, as its core learned behavior rather than a prompted add-on. This is the defining idea behind the class of models often called 'reasoning models' — OpenAI's o1 and o3, DeepSeek's R1, and similar systems that emerged as a distinct model category.

The core shift is in what gets optimized during training. A standard base LLM is trained (via pretraining and typically SFT, from the sft-vs-rft topic) to predict plausible next tokens and follow instructions well — nothing in that training objective specifically rewards GENERATING LONG, EFFECTIVE REASONING before an answer. A reasoning model instead undergoes additional RL-based training (RFT-style, using verifiable rewards on math, code, and logic problems — exactly the GRPO-based approach from sft-vs-rft and rl-environments-for-agents) that directly rewards producing a reasoning trace that leads to a CORRECT final answer. Over enough training, the model learns, on its own, when and how much internal reasoning to generate for a given problem — hard problems trigger much longer internal reasoning; easy ones don't.

This connects directly to 'test-time compute' — the framing that a model's total capability isn't fixed purely by its trained weights, but can be extended by spending more computation AT INFERENCE TIME (i.e., while actually answering a question), rather than only through more TRAINING-time computation. A reasoning model exploits this directly: rather than a fixed-length response, it can generate a much longer internal reasoning trace for a genuinely hard problem, effectively 'spending more thinking time' proportional to the problem's difficulty — this is the same underlying mechanism as CoT (more generated tokens before the answer, from this topic's first subtopic), but now it's the MODEL's own trained judgment deciding how much reasoning to generate, rather than a human prompt requesting a fixed reasoning format.

A distinctive practical detail across many of these models: the internal reasoning trace is often generated but not fully shown to the user in the same way as the final answer — sometimes summarized, sometimes hidden entirely, partly to avoid the model's raw internal deliberation (which may include false starts, self-corrections, or exploratory dead ends) confusing users expecting a clean answer. This differs from CoT prompting a base model, where the full step-by-step trace IS the visible output by design. The practical tradeoff for using reasoning models: meaningfully better performance on genuinely hard reasoning tasks, at the cost of higher latency and cost per query (since more tokens are generated internally before the visible answer) — making them a poor fit for simple queries where a standard model already performs well, and a strong fit specifically for the verifiable-and-hard end of the sft-vs-rft decision tree's task spectrum.

# Illustrating the core behavioral difference: a base model prompted
# with CoT generates a FIXED-shape trace regardless of difficulty; a
# reasoning model's TRAINED behavior adapts reasoning length to
# difficulty automatically -- this is what 'test-time compute' means.

def prompted_cot_model(question: str, difficulty: str) -> dict:
    """A base model given a fixed CoT-style prompt -- reasoning length
    doesn't adapt to the problem's actual difficulty, since a human
    wrote the prompt template, not the model's own judgment."""
    fixed_steps = 3   # the prompt always asks for roughly this much detail
    return {"question": question, "reasoning_steps_generated": fixed_steps}

def trained_reasoning_model(question: str, difficulty: str) -> dict:
    """A model TRAINED (via RL on verifiable rewards) to decide its own
    reasoning length -- more internal reasoning for harder problems,
    learned from training, not requested by a prompt template."""
    difficulty_to_steps = {"trivial": 1, "moderate": 8, "hard": 40}
    steps = difficulty_to_steps.get(difficulty, 5)
    return {"question": question, "reasoning_steps_generated": steps,
            "note": "length chosen by the model's own trained judgment"}

for difficulty in ["trivial", "moderate", "hard"]:
    q = f"a {difficulty} problem"
    prompted = prompted_cot_model(q, difficulty)
    trained = trained_reasoning_model(q, difficulty)
    print(f"{difficulty:10s} | prompted CoT steps: {prompted['reasoning_steps_generated']:3d} "
          f"| reasoning model steps: {trained['reasoning_steps_generated']:3d}")
# The prompted model's step count stays FIXED regardless of difficulty;
# the reasoning model's adapts -- this adaptivity is the core shift