Select Stage — Choosing What Goes Into the Context Window

Where the Write stage is about saving information for later, the Select stage is the complementary act of pulling it back in when it's actually needed. This course refers to this stage as 'Reading context' in its step-by-step breakdown, and as 'Selecting Context' in its summary list — both names describe the same idea: choosing what, out of everything that COULD be included, actually goes into the context window for the current step.

This course identifies three sources this selected context can be pulled from: a tool (calling a live function or API to get current information), memory (retrieving something written to long-term or short-term memory in an earlier step or session), or a knowledge base (docs, a vector DB — the same retrieval mechanism underlying RAG generally, just applied more broadly to whatever context an agent has access to).

The actual decision of what to select isn't arbitrary — it should be driven by relevance (does this piece of information actually bear on the current step's task), recency (more recent information is often, though not always, more useful than stale information), and importance (some facts, like a critical user constraint, deserve inclusion even if they're not the most 'recent' thing available). Getting this selection right is genuinely consequential: pulling in too little context starves the model of information it needs to perform well, while pulling in too much wastes tokens, adds cost and latency, and can actively confuse the model by burying the truly relevant information in noise.

This is exactly the 'LLM is a CPU, context window is RAM' framing elsewhere in this topic: you're not just prompting the model, you're programming what goes into its working memory for this specific step — and just like a real program shouldn't load every byte of available data into RAM 'just in case,' a well-engineered context pipeline selects deliberately rather than including everything available by default.

import numpy as np
from datetime import datetime, timedelta

def select_context(
    candidates: list[dict], query_embedding: np.ndarray, top_k: int = 3,
) -> list[dict]:
    """Score candidate memories by relevance (embedding similarity),
    recency (time decay), and importance (an explicit flag), then keep
    only the top_k — not everything available."""
    def score(item: dict) -> float:
        relevance = float(np.dot(query_embedding, item["embedding"]))
        age_hours = (datetime.now() - item["timestamp"]).total_seconds() / 3600
        recency = 0.5 ** (age_hours / 24)  # decays over ~24h half-life
        importance_boost = 0.3 if item.get("important") else 0.0
        return relevance * 0.6 + recency * 0.3 + importance_boost

    scored = sorted(candidates, key=score, reverse=True)
    return scored[:top_k]

candidates = [
    {"text": "User's subscription tier is Pro", "embedding": np.array([0.9, 0.1]),
     "timestamp": datetime.now() - timedelta(hours=1), "important": True},
    {"text": "User mentioned liking the color blue once", "embedding": np.array([0.1, 0.9]),
     "timestamp": datetime.now() - timedelta(days=10), "important": False},
]
selected = select_context(candidates, query_embedding=np.array([0.85, 0.15]))