Compression and Selection: Filtering Aggregated Context Before Generation

The previous subtopic's four retrieval sources (documents, memory, web, arXiv) each hand back their own results independently — but simply concatenating all of them together and handing the whole pile to the response-generating agent would run straight into the Compressing-stage problem from the companion topic: duplicate or redundant information, leading to extra tokens and increased cost. This is exactly what this course's step #7 addresses.

Step #7, Filter context, per this course: now, we pass our combined context to the context evaluation agent that filters out irrelevant context. This filtered context is then passed to the synthesizer agent that generates the final response. Two things are worth noticing about this design choice. First, filtering is done by a DEDICATED agent, not a hand-written rule or a simple keyword filter — this course uses an LLM-powered agent specifically BECAUSE relevance judgment (is this arXiv paper actually useful for answering this specific query, versus just topically related) is a nuanced task better suited to an LLM's judgment than to brittle heuristics. Second, filtering happens as its OWN separate step, cleanly decoupled from the final response generation — the context-evaluation agent's only job is deciding what stays and what goes; it doesn't also try to write the final answer, keeping each agent's responsibility narrow and focused.

This connects directly to the companion topic's Compressing stage definition — 'keeping only the tokens needed for a task' — but shows a genuinely different MECHANISM than the simpler 'summarize the text' approach that subtopic mentioned. Rather than compressing each individual piece of context down to fewer tokens (lossy summarization), this workflow's filtering step makes a binary keep/drop decision on WHOLE items — an entire irrelevant document chunk or arXiv result gets dropped entirely, rather than being shrunk. Both are valid Compressing techniques; which one fits depends on whether your problem is 'too many irrelevant items' (favors filtering/selection) or 'individually-relevant items that are each too verbose' (favors summarization).

The practical result of this design: by the time context reaches the synthesizer agent (next subtopic's final step), it's already been narrowed from potentially four sources' worth of raw results down to just the pieces genuinely relevant to answering this specific query — directly controlling both the token cost and, just as importantly, the RISK of the final response being distracted or diluted by irrelevant retrieved context.

# Step #7: an LLM-agent-style context filter -- a binary keep/drop
# decision per item, contrasted with per-item summarization compression.

def context_evaluation_agent(aggregated_context: list[str], query: str) -> list[str]:
    """Stand-in for the book's LLM-powered filtering agent -- makes a
    keep/drop judgment per item based on relevance to the query.
    A real implementation would call an LLM to judge relevance;
    here a simple relevance heuristic illustrates the SHAPE of the step."""
    query_terms = set(query.lower().split())
    filtered = []
    for item in aggregated_context:
        item_terms = set(item.lower().split())
        relevance = len(query_terms & item_terms) / max(len(query_terms), 1)
        if relevance > 0.15:          # keep only genuinely relevant items
            filtered.append(item)
    return filtered

def summarize_compress(item: str, max_words: int = 6) -> str:
    """An ALTERNATIVE compression mechanism -- shrink an individually-
    relevant item, rather than dropping it entirely."""
    words = item.split()
    return " ".join(words[:max_words]) + ("..." if len(words) > max_words else "")

aggregated = [
    "attention mechanism computes weighted sums over value vectors",
    "the weather in paris is sunny today",                              # irrelevant, gets dropped
    "transformer attention was introduced in the Attention Is All You Need paper",
]
query = "how does attention work"

filtered = context_evaluation_agent(aggregated, query)
print(f"Kept {len(filtered)} of {len(aggregated)} items after filtering:")
for item in filtered:
    print("  -", item, " -> summarized:", summarize_compress(item))