In-Context Memory: What Fits in the Context Window

The most basic — and most immediately available — form of agent memory doesn't require any external system at all: it's simply whatever information currently sits inside the LLM's context window for the active request. Conversation history, documents retrieved earlier in the same session, and tool outputs from previous steps in an agent loop are all forms of in-context memory, as long as they're still present in the current prompt being sent to the model.

This is the mechanism behind the Agent class pattern seen elsewhere in this curriculum: appending every user message and assistant reply to a running self.messages list, and sending that entire list with every API call. As long as that history stays inside the context window, the model behaves as if it 'remembers' everything in it — it doesn't actually remember anything between API calls (LLMs are stateless), it's just being re-shown the same history each time.

In-context memory's core appeal is that it's essentially free and requires zero additional infrastructure — no database, no vector store, nothing beyond what you're already sending in the prompt. It's also immediately and perfectly accurate, since the model is reading the exact original text, not a summary or a retrieved approximation of it.

The catch is the same thing that makes it simple: it's strictly bounded by the context window's size, and it evaporates completely the instant the session ends or the history grows too large to keep including in full. A long-running conversation, or an agent that's executed many tool calls, will eventually generate more history than fits in the window — at which point something has to give: either older messages get dropped, or the history needs compressing (via summarization), or it needs to move into some form of persistent, external memory instead — which is exactly the gap external memory (the next subtopic) is designed to fill.

class InContextMemoryAgent:
    def __init__(self, system: str, max_context_tokens: int = 8000):
        self.messages = [{"role": "system", "content": system}]
        self.max_context_tokens = max_context_tokens

    def _approx_tokens(self) -> int:
        # Rough estimate: ~4 chars per token — real code uses a tokenizer
        return sum(len(m["content"]) for m in self.messages) // 4

    def add(self, role: str, content: str) -> None:
        self.messages.append({"role": role, "content": content})
        if self._approx_tokens() > self.max_context_tokens:
            # Simplest strategy: drop the oldest non-system messages —
            # a real system would summarize instead of just dropping
            while self._approx_tokens() > self.max_context_tokens and len(self.messages) > 1:
                self.messages.pop(1)  # keep index 0 (system prompt)

agent = InContextMemoryAgent(system="You are a helpful assistant.")
agent.add("user", "My order number is 12345.")
agent.add("assistant", "Got it, I'll reference order 12345.")
# Everything the model "remembers" right now is exactly what's in self.messages