In-Context Memory: What Fits in the Context Window
~12 min read
The simplest form of agent memory is just what's currently sitting in the prompt — conversation history, retrieved documents, tool outputs. It's fast and free of infrastructure, but it's bounded by the context window and disappears the moment the session ends.
In-Context Memory: What Fits in the Context Window is a Pro topic
Sign in, then upgrade to Pro or Power to unlock this topic and the full AI Engineering curriculum.
Key points
- •In-context memory is just what's currently inside the LLM's prompt — conversation history, retrieved docs, prior tool outputs
- •LLMs are stateless between calls — 'remembering' is really just re-sending the same history with every request
- •Free and perfectly accurate — no infrastructure needed, and the model reads the exact original text
- •Strictly bounded by context window size, and disappears completely once the session ends
- •Long conversations or many tool calls eventually force a choice: drop old messages, summarize/compress, or move to external memory