Conversational Memory
Techniques for persisting context across multiple conversation turns in LLM chat applications: full buffer, windowed buffer, summary memory, and entity memory — and how to choose between them given context window constraints and conversation length.
▶📚 Prerequisites(2)
🎓 Learning objectives
- •Explain why LLMs are stateless and what that means for multi-turn conversations
- •Implement conversation buffer memory and identify when it exceeds context limits
- •Apply windowed buffer memory with the correct window size for a given use case
- •Implement summary memory that compresses old conversation history while preserving key facts
- •Choose the right memory strategy given context window size, conversation length, and what needs to be remembered
What is it?
Conversational memory refers to techniques for managing chat history across multiple turns so that an LLM can respond with awareness of what was said earlier in the conversation. Since LLMs are fundamentally stateless — each API call is independent and the model has no built-in memory across calls — the application layer is responsible for deciding what past context to include in each new request.
The core trade-off in every memory strategy is between context coverage (how much of the conversation history the model can see) and context cost (tokens are finite and expensive). Chip Huyen (AI Engineering, Ch.8) frames this as the 'conversation context management' problem: what to keep, what to compress, and what to discard from the growing chat history as a conversation progresses.
Why it exists
LLMs process one context window per API call. The model has no persistent state between calls — it doesn't 'remember' turn 1 when answering turn 20 unless you explicitly include turn 1 in the turn 20 request. This is unlike human memory, which naturally accumulates across a conversation.
For short conversations, this is fine — you include all previous turns and the context fits in the window. But for longer conversations:
- 100 turns × 200 tokens/turn = 20,000 tokens just for history
- At 128K context, this is fine; at 4K context, it overflows after 20 turns
- Even with large context windows, more tokens = higher cost and slower inference
Conversational memory strategies are the solutions: instead of dumping all history into every context, they selectively retain the most useful prior context in a token-efficient form.
Problem it solves
- After 30 conversation turns, my chatbot 'forgets' what the user said at turn 1 — how do I fix this?
- My context window fills up after 10 turns and the API throws a context-length error — how do I handle long conversations?
- I want the chatbot to remember user preferences mentioned early in the conversation without keeping every word of the chat history.
- How is conversational memory different from RAG? When should I use one vs. the other?
- The model costs too much because I'm including the full chat history in every request — how do I reduce token usage while preserving key context?
Intuition
Conversational memory is like a meeting note-taker.
Imagine you're in a 3-hour client meeting. You don't read every word of the transcript to answer a question at hour 3 — that would take too long and cost too much attention. Instead, you use a layered approach:
- The last 10 minutes you remember perfectly (windowed buffer)
- The first 2 hours exist as your meeting notes — a summary of key decisions, action items, and context (summary memory)
- Specific facts ('client's budget is $500K', 'launch date is Q3') you've written down explicitly and can look up instantly (entity memory)
This is exactly how well-designed conversational memory systems work: recent turns are kept verbatim, older turns are compressed into summaries, and critical entities are extracted and maintained separately. The full transcript is available but rarely needed.
Analogy
Conversational memory strategies map directly to human working memory limitations.
Your working memory (what you actively hold in mind) is small — Miller's Law says ~7 items. For a long meeting, you actively hold the recent few exchanges, rely on summarized notes for earlier parts, and check your CRM for specific facts. You don't try to hold everything in working memory simultaneously.
LLM context windows are a formal version of working memory: finite, expensive to fill, and the most recent content gets the most attention. The analogy maps to memory strategies:
- Full buffer = trying to hold every word of the meeting in working memory (fails quickly)
- Windowed buffer = keeping only the last 10 minutes in working memory
- Summary memory = summarizing older content into a 1-page brief before the meeting ends
- Entity memory = maintaining a CRM record for specific facts that need to persist regardless
Technical explanation
MEMORY STRATEGY 1 — Full Conversation Buffer: Store all messages; pass the full list to the API on every turn. messages = [(role, content), ...] messages.append((new_role, new_content)) response = client.messages.create(model=..., messages=messages) When to use: conversations expected to be < 20-30 turns and context window is large. When it fails: after 50+ turns in a 4K window, or 200+ turns in a 128K window.
MEMORY STRATEGY 2 — Windowed Buffer: Keep only the last K turns. Simple slice. recent_messages = messages[-K*2:] # K turns = 2K messages (user+assistant pairs) When to use: conversations that only need recent context (customer support, simple Q&A). Limitation: if the user mentions their name in turn 1 and asks about it in turn 51, a K=25 window will fail. Losing the system prompt is a common bug — keep it separately.
MEMORY STRATEGY 3 — Summary Memory: When the buffer exceeds a token threshold, compress the oldest N turns into a summary:
- Detect: token_count(messages) > threshold
- Take the oldest chunk: chunk = messages[:N]
- Summarize: summary = llm.summarize(chunk) # 'User mentioned X, discussed Y...'
- Replace: messages = [system_with_summary] + messages[N:]
- Or prepend: messages = [summary_message] + messages[N:] When to use: long conversations where gist is enough (therapy chatbot, tutoring). Limitation: once summarized, specific wording is lost — the model can reason about the summary but cannot quote the original.
MEMORY STRATEGY 4 — Entity Memory: Extract structured facts from the conversation and maintain them as a key-value store: entities = {} # {'name': 'Alice', 'budget': '$500K', 'deadline': 'Q3'}
After each turn: parse for entity updates
Inject into context: f'Known facts: {json.dumps(entities)}'
When to use: when specific facts must persist perfectly across many turns. Common in: CRM assistant, personal finance chatbot, medical intake bot.
HYBRID APPROACHES: Most production systems combine strategies:
- Recent buffer (last 10 turns) for immediate context
- Summary for older turns
- Entity store for structured facts
- RAG for external document retrieval (note: RAG ≠ memory — it retrieves from a knowledge base, not from conversation history)
DISTINGUISHING FROM RELATED CONCEPTS:
- Conversational memory: manages CONVERSATION HISTORY across turns (same session)
- RAG: retrieves from an EXTERNAL KNOWLEDGE BASE (documents, database)
- Context engineering: optimizes a SINGLE context window (one API call, not cross-turn) These are complementary, not competing. A production system may use all three.
Architecture
Production Conversational Memory System:
┌─────────────────────────────────────────────────────────────┐ │ Chat Session State (per user, per session) │ │ │ │ entity_store: {name, preferences, facts, constraints} │ │ summary: 'In earlier conversation: user discussed X...' │ │ recent_turns: [last 10-20 messages verbatim] │ └──────────────────────────────┬──────────────────────────────┘ │ ▼ On each new user message:
┌─────────────────────────────────────────────────────────┐ │ 1. CONTEXT ASSEMBLY │ │ [system_prompt] │ │ + [entity_store as JSON block] │ │ + [summary_message] (if exists) │ │ + [recent_turns] │ │ + [new user message] │ └───────────────────────┬─────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ 2. LLM CALL → response │ └───────────────────────┬─────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ 3. POST-TURN UPDATE │ │ a. Append (user, response) to recent_turns │ │ b. Extract new entities from the turn │ │ c. If recent_turns > threshold: │ │ - Summarize oldest N turns → update summary │ │ - Drop oldest N from recent_turns │ └─────────────────────────────────────────────────────────┘
Workflow
-
ESTIMATE expected conversation length and content:
- Short (< 20 turns, simple Q&A): full buffer is fine
- Medium (20-100 turns, customer support): windowed buffer
- Long (100+ turns, tutoring, therapy): summary + recent window
- Facts that MUST persist across many turns: entity memory
-
IMPLEMENT the chosen strategy:
- Always keep the system prompt separately (never include in the sliding window)
- Track token count after every turn (use tokenizer, not word count)
- Set trigger threshold at 80% of context limit (leave headroom for response)
-
TEST the memory behavior:
- Simulate a 50-turn conversation
- Check: does the model remember turn 1 facts at turn 50?
- Check: what happens at exactly the window/summary trigger threshold?
- Common bug: system prompt gets dropped when windowing
-
MONITOR token usage in production:
- Log input_tokens per turn; alert if approaching context limit
- Track cost per conversation (longer memory = higher cost)
-
DECIDE on persistence scope:
- In-memory (dict): simplest, lost on server restart
- Redis: fast, persistent, good for session state
- Database: full persistence, queryable, right for long-lived users
Example
from anthropic import Anthropic import tiktoken client = Anthropic() class SummaryMemoryChat: '''Chat with windowed buffer + summary compression for long conversations.''' def __init__( self, model: str = 'claude-sonnet-4-6', system: str = 'You are a helpful assistant.', max_tokens: int = 1000, token_threshold: int = 3000, # compress when history > 3K tokens turns_to_keep: int = 10, # keep last 10 turns verbatim after compress ): self.model = model self.system = system self.max_tokens = max_tokens self.token_threshold = token_threshold self.turns_to_keep = turns_to_keep self.messages: list[dict] = [] self.summary: str = '' def _token_count(self) -> int: # Approximate — production should use tiktoken or model's count_tokens total = sum(len(m['content'].split()) * 1.3 for m in self.messages) return int(total) def _compress(self) -> None: '''Summarize oldest half of messages and update self.summary.''' keep_n = self.turns_to_keep * 2 # turns_to_keep turns × 2 messages/turn to_summarize = self.messages[:-keep_n] if not to_summarize: return history_text = '\n'.join( f"{m['role'].upper()}: {m['content']}" for m in to_summarize ) existing = f'Previous summary: {self.summary}\n\n' if self.summary else '' prompt = ( f'{existing}Compress the following conversation into a concise paragraph ' f'preserving all important facts, decisions, and context:\n\n{history_text}' ) resp = client.messages.create( model=self.model, max_tokens=300, messages=[{'role': 'user', 'content': prompt}], ) self.summary = resp.content[0].text self.messages = self.messages[-keep_n:] # keep only recent def chat(self, user_input: str) -> str: # Compress if approaching context limit if self._token_count() > self.token_threshold: self._compress() self.messages.append({'role': 'user', 'content': user_input}) # Build context: system + optional summary + recent messages system_content = self.system if self.summary: system_content += f'\n\n[Earlier conversation summary: {self.summary}]' response = client.messages.create( model=self.model, max_tokens=self.max_tokens, system=system_content, messages=self.messages, ) reply = response.content[0].text self.messages.append({'role': 'assistant', 'content': reply}) return reply # Usage chat = SummaryMemoryChat() print(chat.chat('My name is Alice and my budget is $500K.')) # ... many turns later ... print(chat.chat('What was my budget?')) # recalled from summary
Real-world usage
-
Customer support chatbots (Intercom, Zendesk AI): use windowed buffer (last 10-20 turns) since support sessions are bounded. Key user data (account ID, issue category) extracted as entities at session start and always retained.
-
OpenAI ChatGPT's memory feature (2024): stores explicit user-declared facts (entity memory) across sessions — 'remember that I prefer short answers' — while keeping recent turns as conversation buffer within a session.
-
LangChain ConversationSummaryBufferMemory: the canonical OSS implementation of the hybrid pattern: token threshold triggers summary compression of older turns, recent K turns kept verbatim. Used in thousands of production LLM applications.
-
Chip Huyen (AI Engineering, Ch.8): 'The choice of memory strategy depends on what the conversation needs to remember. Most production systems use a hybrid: entity store for structured facts, summary for older history, buffer for recent context.'
-
Jay Alammar (Hands-On LLMs, Ch.7): provides worked implementations of full buffer, windowed buffer, and summary memory patterns, noting that windowed buffer is the most common production choice due to its simplicity and bounded cost.
Trade-offs
Recall vs. cost: full buffer has perfect recall but unbounded cost. Summary memory has bounded cost but imperfect recall. Entity memory has precise recall for specific facts but only works for structured information. Choose based on what matters: for support chatbots, recent context is enough; for personal assistants, entity memory for preferences matters; for tutoring, summary of the learning journey matters.
Latency vs. memory quality: summarization adds an extra API call per compression event, adding latency. In high-volume applications, this can be significant. Alternative: use a fast, cheap model (Haiku/GPT-4o-mini) for summarization.
Simplicity vs. capability: windowed buffer takes 3 lines of code and costs nothing extra. Hybrid memory takes 200+ lines and adds operational complexity. Start with windowed buffer and add complexity only when users complain about the model forgetting things it should remember.
Visual explanation
Stateless LLM — why memory is needed:
Turn 1: [system] [user: 'My name is Alice'] → [assistant: 'Hello Alice!'] Turn 2: [system] [user: 'What is 2+2?'] → [assistant: '4'] Turn 3: [system] [user: 'What is my name?'] → [assistant: 'I don't know your name'] ↑ stateless — no memory!
With conversation buffer: Turn 3 request: [system] [user: 'My name is Alice'] [assistant: 'Hello Alice!'] [user: 'What is 2+2?'] [assistant: '4'] [user: 'What is my name?'] → [assistant: 'Your name is Alice.']
Memory Strategy Comparison:
FULL BUFFER (keep everything): [t1][t2][t3]...[t100] → context ✓ Perfect recall ✗ Grows without bound ✗ Expensive at scale
WINDOWED BUFFER (keep last K turns): [t91][t92]...[t100] → context (K=10) ✓ Bounded cost ✓ Simple ✗ Forgets everything before window
SUMMARY MEMORY (compress old turns): [SUMMARY of t1-t90] + [t91]...[t100] → context ✓ Preserves gist of old context ✗ Summary loses detail ✗ Summarization cost
ENTITY MEMORY (extract key facts): [ENTITIES: name=Alice, budget=$500K, deadline=Q3] + [t91]...[t100] → context ✓ Precise recall of critical facts ✗ Only works for structured facts ✗ Extraction accuracy
Advantages
- —
Summary memory dramatically reduces token cost for long conversations while preserving the semantic gist of earlier turns
- —
Entity memory enables perfect recall of critical structured facts (names, preferences, constraints) across arbitrarily long conversations
- —
Windowed buffer is trivially simple to implement and has fully predictable, bounded context cost
- —
Hybrid approaches give you the best of all strategies: precision for critical facts, efficiency for recent context, compression for older history
- —
The right memory strategy can reduce API costs by 60-90% compared to full conversation buffer in long conversations
Disadvantages
- —
Summary memory loses exact wording — the model can reason about summarized content but cannot quote or recall specific phrasing from compressed turns
- —
Entity memory requires accurate extraction — if the entity extractor misses a key fact or extracts it incorrectly, that error persists
- —
Windowed buffer simply forgets everything before the window — not suitable for tasks where early context is critical
- —
Every summarization call adds latency and API cost, which can be significant in applications with many concurrent long conversations
- —
Context management bugs (system prompt getting dropped in windowing, summary overwriting entity store) are subtle and hard to catch in unit tests
Common mistakes
- —
Dropping the system prompt during windowing. A common bug:
messages[-K:]accidentally excludes the first message (system prompt). Always store the system prompt separately and prepend it explicitly to every API call, regardless of the memory strategy. - —
Counting words instead of tokens. Memory strategies must trigger based on token count, not word count or character count. English text is ~1.3 tokens/word on average, but code and non-English text can be 2-5 tokens/word. Use the model's tokenizer or the API's
count_tokensmethod for accurate measurement. - —
Confusing conversational memory with RAG. RAG retrieves from an external knowledge base (documents, database). Conversational memory manages the conversation history within a session. They're complementary: a customer support bot might use RAG for product documentation and summary memory for the current support ticket's conversation.
- —
Setting the compression threshold too high. If you compress only when you're at 99% context capacity, the compress call might itself overflow the context. Set the threshold at 70-80% to leave headroom for the summarization prompt and response.
- —
Using the same model for summarization as for the main conversation. A GPT-4 Turbo summary call costs the same as a main conversation turn. Use a fast, cheap model (GPT-4o-mini, Claude Haiku) for summarization — the task is simple enough that the quality difference is negligible.
🎤 Interview questions
Why are LLMs stateless, and what does that mean for building a multi-turn chatbot? Describe three memory strategies and when you'd choose each.
How is conversational memory different from RAG? Give a scenario where you'd use both in the same system.