Prompt Engineering & Chain-of-Thought Reasoning — AI Engineering

Prompt reasoning encompasses the techniques that improve LLM reasoning quality: Chain-of-Thought (CoT) prompting that elicits step-by-step reasoning, few-shot examples that demonstrate problem-solving patterns, and structured output formats that guide the model's response shape. Reasoning models like o1 and o3 internalize CoT as a training objective, producing longer but more accurate outputs.

Chain-of-Thought techniques:

1. Zero-shot CoT: append 'Let's think step by step' to any question. This single phrase improves accuracy on multi-step problems by 30–40% on benchmarks like GSM8K (math word problems).

2. Few-shot CoT: include 3–8 examples showing [question, reasoning trace, answer]. The model learns the reasoning format from examples. More powerful than zero-shot but consumes context budget.

3. Self-consistency: generate 5–10 independent CoT reasoning paths, take majority vote on final answer. Reduces variance, improves accuracy on ambiguous problems. 5× more expensive but more reliable.

4. Tree-of-Thought (ToT): explore multiple reasoning branches simultaneously, evaluate each branch, prune low-quality paths. Best for problems with discrete solution space (puzzles, planning).

Structured outputs: instruct the model to respond in a specific format. Use XML tags for sections (......) — models parse and follow tag-based structures more reliably than plain instructions.

# Zero-shot CoT (simplest, often enough)
response = client.messages.create(
    model="claude-opus-4-5",
    messages=[{"role": "user", "content":
        "A train leaves NYC at 9am going 60mph. Another leaves Boston (215 miles away) at 10am going 80mph. When do they meet? Think step by step."
    }]
)

# Self-consistency (majority vote over 5 paths)
from collections import Counter
answers = []
for _ in range(5):
    r = client.messages.create(model="claude-opus-4-5",
        messages=[{"role": "user", "content": question + " Think step by step."}])
    answers.append(extract_final_answer(r.content[0].text))
final = Counter(answers).most_common(1)[0][0]