Context Engineering: Memory & History
Learn context assembly, the 6 types of contexts for AI agents, memory hierarchies, token compression, and context engineering workflows.
Semantic paragraph chunking
PDF textbooks are parsed and split into chunks with 100-character overlaps to keep semantic continuity.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Identify and implement the 6 Types of Contexts for AI Agents (Instructions, Examples, Knowledge, Memory, Tools, Tool Results)
- •Optimize context windows using token compression and summarization
- •Design context engineering workflows utilizing tools like Tensorlake, Zep, Firecrawl, and Milvus
What is it?
Context engineering is the discipline of designing, managing, and optimizing what goes into an LLM's context window to maximize output quality and minimize cost. While "prompt engineering" refers to wording individual prompts, context engineering covers the full architecture: system prompts, conversation history management, tool outputs, retrieved documents, memory hierarchies, and token budget allocation.
Why it exists
Context windows are finite and expensive. GPT-4 at 128K context costs $1.28 per full-context request. Most of that budget is wasted on redundant history, verbose tool outputs, or generic system prompts. Context engineering emerged as teams realized that what you put in the window determines output quality as much as model choice does.
Problem it solves
Three concrete problems: (1) context overflow — important information from earlier in a conversation gets pushed out of the window; (2) context pollution — irrelevant or verbose content dilutes the signal, degrading response quality; (3) cost explosion — unmanaged conversation history grows linearly, making production costs unsustainable.
Intuition
Think of the context window as desk space for your AI assistant. If their desk is cluttered with old emails, random notes, and irrelevant documents, they will miss the important memo right in front of them. Context engineering is keeping the desk clean: removing old irrelevant items, highlighting the most important documents, and ensuring the current task instructions are visible at all times.
If you come from Java/Spring Boot: context management is like request-scoped dependency injection. You carefully decide what data to inject into each request handler — not the entire application state. Context engineering decides what to inject into each LLM call, using patterns analogous to @RequestScope vs @SessionScope vs @ApplicationScope.
If you come from React/Frontend: context engineering is like Redux state management. You decide which state goes in global store (system prompt / persona), which in component state (current turn), and which gets fetched on demand (retrieved documents via RAG). You do not put the entire application state into every component's props.
Analogy
Context engineering is like prepping a briefing document for a high-stakes executive meeting. You do not hand the executive 10,000 pages of company data — you distill the 2-page brief with exactly what they need: the decision to make, the 3 most relevant data points, the recommended action. The executive (LLM) makes better decisions from a curated brief than from an overwhelming raw data dump.
Technical explanation
Context window anatomy: System prompt: persona, instructions, constraints, output format (typically 500–2000 tokens) Conversation history: alternating user/assistant turns (grows unbounded in naive implementations) Tool definitions: JSON schemas for available tools (100–500 tokens per tool) Retrieved documents: RAG context (500–5000 tokens) Current user message: actual query (50–500 tokens)
Cost example (GPT-4, $10/1M input tokens): 20-turn conversation × 500 tokens/turn = 10K tokens history 5 tool schemas × 200 tokens = 1K tokens 3 retrieved chunks × 600 tokens = 1.8K tokens Total per request: ~12.8K tokens × $10/1M = $0.128 per request At 10K requests/day: $1,280/day just for input tokens
Context management strategies:
- Sliding window: keep only last N turns in history
- Summarization: replace older turns with LLM-generated summary (recursive compression)
- Selective retrieval: only include conversation turns relevant to current query (embedding-based)
- Tiered memory: hot (current window) → warm (summary) → cold (vector store)
Prompt caching (Anthropic): prefix cache hits return at 10% of full cost. Cache the static parts (system prompt + tool schemas + boilerplate instructions) and only pay full price for the dynamic parts (current turn + retrieved docs). Typical savings: 60–80% on repeated-prefix workloads.
Architecture
Context window allocation (128K token budget):
[System Prompt] 2K tokens — static, cache-eligible [Tool Schemas] 1K tokens — static, cache-eligible [Conversation Summary] 1K tokens — updated every N turns [Recent Turns] 5K tokens — last 5–10 turns verbatim [Retrieved Context] 4K tokens — per-query RAG results [Current User Message] 0.5K tokens [Reserved for Output] 20K tokens ───────────────────────────────────── Total used: 33.5K of 128K
Memory hierarchy: L1: Context window (immediate, expensive) L2: Summary buffer (compressed, cheap to maintain) L3: Vector store (semantic search, pennies per query) L4: Relational DB / key-value store (structured facts)
Workflow
- Audit current context composition: log token counts for each component (system prompt, history, tools, retrieved docs)
- Identify waste: what is in the context that is not helping the current response?
- Apply sliding window: limit conversation history to last 8–10 turns
- Implement progressive summarization: after every 10 turns, summarize the conversation so far and replace the turns with the summary
- Cache static prefixes: move system prompt + tool schemas to a cached prefix (Anthropic/OpenAI support this)
- Right-size RAG: retrieve 3–5 chunks, not 20 — more is not better
- Compress tool outputs: truncate verbose API responses to the 2–3 relevant fields
- Monitor: track p50/p95 input token counts per request in production
Example
Conversation history manager with sliding window + summarization
class ContextManager: def init(self, max_turns: int = 8, summarize_after: int = 12): self.turns = [] # recent conversation turns self.summary = "" # compressed older history self.max_turns = max_turns self.summarize_after = summarize_after
def add_turn(self, role: str, content: str):
self.turns.append({"role": role, "content": content})
if len(self.turns) > self.summarize_after:
self._compress()
def _compress(self):
# LLM call to summarize oldest half of turns
old_turns = self.turns[:self.summarize_after // 2]
self.summary = summarize_with_llm(old_turns, self.summary)
self.turns = self.turns[self.summarize_after // 2:]
def get_messages(self) -> list:
prefix = [{"role": "user", "content": f"Conversation summary: {self.summary}"},
{"role": "assistant", "content": "Understood."}] if self.summary else []
return prefix + self.turns[-self.max_turns:]
Real-world usage
LangChain ConversationSummaryBufferMemory: implements progressive summarization — maintains a running LLM-generated summary of older turns plus a verbatim buffer of recent turns. Used in production chatbots handling 100-turn+ conversations.
Anthropic prompt caching: major enterprise customers (Notion, Slack) cache multi-thousand-token system prompts, reducing costs by 70–80% on their highest-volume endpoints. The cache persists for 5 minutes with a TTL refresh on each cache hit.
Cursor IDE context management: automatically selects relevant files, function definitions, and cursor position context to include in each coding prompt. The quality of Cursor's context selection is a core competitive advantage over simpler tools.
Trade-offs
Context length vs cost: every token in context costs money on every request. Doubling context length doubles input cost. Most production systems never need 128K tokens — design for 4K–16K and only expand when necessary.
Summarization quality vs fidelity: LLM-based compression loses specific details (exact numbers, names, dates). For tasks requiring precise recall of specific earlier statements, keep full verbatim history instead of summarizing. Use summarization for conversational context, not factual records.
Cache hit rate vs freshness: caching the system prompt saves money but means you cannot personalize it per-user. Multi-tenant systems must choose: one cached generic system prompt (cheaper) vs per-user dynamic prompts (more personalized, no cache benefit).
Visual explanation
Structured Prompt Caching Scheduling: ┌─────────────────────────────────────────────────────────┐ │ LLM CONTEXT WINDOW │ ├─────────────────────────────────────────────────────────┤ │ [Static System Instructions] <── (Cached Block - Cheap)│ │ [Static Tool Schemas] <── (Cached Block - Cheap)│ │ [Episodic / Long-term Memory] <── (Retrieved dynamically)│ │ [Compressed History] <── (Dynamic sliding window)│ │ [User Query / Input] <── (Trigger generation) │ └─────────────────────────────────────────────────────────┘
Advantages
- —
Saves up to 60% token charges via compression
- —
Reduces model confusion and increases execution accuracy
Disadvantages
- —
Compression steps introduce minor pipeline latency
Common mistakes
- —
Never clearing conversation history in a long-running session. A 100-turn chat session accumulates 50K+ tokens of history — every subsequent turn must process the entire history. This makes late-conversation responses expensive (100× the cost of turn 1) and eventually hits the context limit.
- —
Dumping full API responses as tool results. A web search returning 10K characters of HTML, a database query returning 500 rows — these verbose tool outputs consume enormous context budget. Always truncate and summarize tool outputs before including them: "Search found 847 results. Top 3: [concise summaries]".
- —
Treating all conversation turns as equally important. The first 2 turns (initial request + clarification) and the last 3 turns (current context) are far more important than turns 5–15. Implement recency-biased summarization that compresses the middle while preserving the beginning and end.
- —
Not measuring context token usage in development. Teams discover context issues in production when costs spike unexpectedly. Log input_tokens from every API response during development and test with realistic conversation lengths, not 3-turn demos.
- —
Putting dynamic content before static content in the prompt. Prompt caching works by matching a static prefix. If your system prompt is dynamic (includes today's date, user name, or live data), nothing after it can be cached. Move all dynamic content to the end of the prompt, after the static cacheable portion.
🎤 Interview questions
Explain prompt caching. How does the arrangement of static and dynamic elements in a prompt affect API serving latency and cost?
📂 Subtopics
Write Stage — Creating Context: Memory, Retrieved Docs, Tool Results, History
Writing context means saving information OUTSIDE the active context window so it can help an agent perform a task later — to long-term memory, short-term memory, or a state object, rather than trying to keep everything live at once.
~12 min
Select Stage — Choosing What Goes Into the Context Window
Selecting (the book also calls this 'Reading') context means pulling previously-written information back INTO the context window when it's actually needed — from a tool, from memory, or from a knowledge base — based on relevance, recency, and importance.
~12 min
Compress Stage — Making Context Smaller: Summarization, Pruning, Distillation
Compressing context means keeping only the tokens actually needed for the task at hand. Retrieved context and multi-turn tool-call history often contain duplicate or redundant information that inflates token count and cost — summarization is the main fix.
~12 min
Isolate Stage — Separating Context by Type So the Model Doesn't Confuse Instructions, Data, and History
Isolating context means splitting it up rather than dumping everything into one undifferentiated blob — via multiple agents each with their own scoped context, a sandbox for code, or a state object — so the model doesn't confuse what's an instruction, what's data, and what's history.
~12 min