Compress Stage — Making Context Smaller: Summarization, Pruning, Distillation

Even with careful selection (the previous stage), the context that gets pulled in can still be larger than it needs to be — the Compress stage is the third of the 4 fundamental stages, and it means keeping only the tokens actually needed for the task at hand, rather than the full raw text of everything that was selected.

This course is specific about why this matters in practice: the retrieved context may contain duplicate or redundant information, particularly across multi-turn tool calls — an agent that's called the same lookup tool three times across a conversation might have three near-identical chunks of retrieved text sitting in its history, each one costing real tokens without adding real information. This directly leads to extra tokens and increased cost, on top of eating into the context window's finite capacity that could otherwise hold genuinely new information.

Context summarization is the main technique this course points to for solving this: rather than keeping the full raw text of, say, 10 prior tool calls and their complete results, an agent can periodically summarize that history into a denser representation — 'searched for X, Y, and Z; found relevant results for X and Z; Y returned nothing useful' — that captures the useful signal in a fraction of the tokens. This is a form of lossy compression: some detail genuinely gets discarded, which is an acceptable trade-off specifically because the discarded detail (the raw duplicate phrasing, the redundant restatements) wasn't adding decision-relevant value in the first place.

Beyond summarization specifically, the same goal — fewer tokens, same useful signal — can be pursued via pruning (dropping clearly irrelevant or superseded content entirely, rather than summarizing it) and distillation (extracting just the key facts or conclusions from a longer piece of context, discarding the reasoning or narrative that led to them). All three techniques share the same underlying goal this stage is built around: the context window should carry the maximum useful signal per token, not the maximum raw information regardless of cost.

from openai import OpenAI

client = OpenAI()

def compress_tool_history(tool_calls: list[dict], max_tokens_estimate: int = 300) -> str:
    """When accumulated tool-call history gets too large, summarize it
    into a denser form instead of keeping every raw result verbatim."""
    raw_history = "\n".join(f"{tc['tool']}({tc['args']}) -> {tc['result']}" for tc in tool_calls)

    # Rough token estimate — real code uses an actual tokenizer
    if len(raw_history) // 4 <= max_tokens_estimate:
        return raw_history  # already small enough — no compression needed

    resp = client.chat.completions.create(model="gpt-4.1", messages=[
        {"role": "user", "content":
            f"Summarize this tool-call history into a dense list of what was "
            f"tried and what was found, dropping redundant/duplicate results:\n\n{raw_history}"}
    ])
    return resp.choices[0].message.content  # same useful signal, far fewer tokens

history = [
    {"tool": "search_docs", "args": "pricing", "result": "Found pricing page v1..."},
    {"tool": "search_docs", "args": "pricing plans", "result": "Found pricing page v1 (same as above)..."},
    {"tool": "search_docs", "args": "enterprise pricing", "result": "No results found."},
]
print(compress_tool_history(history))