Memory Types & Architecture for AI Agents
How agents remember — semantic/episodic/procedural long-term memory types, why memory-less agents are a blank slate every interaction, and the 5-part memory architecture (short-term, long-term, entity, contextual, user).
Formulate task objective
Task: Fetch active users and generate database summary report.
▶📚 Prerequisites(1)
🎓 Learning objectives
- •Distinguish semantic, episodic, and procedural long-term memory in agents, paralleling human memory types
- •Explain why a memory-less agentic system is a 'blank slate' on every interaction
- •Describe the 5-part memory architecture (short-term, long-term, entity, contextual, user memory) and what each stores
- •Recognize memory as a system-design problem, not a property of the LLM itself
What is it?
Agent memory is the mechanism by which an agent retains and reuses information across interactions instead of treating every message as the first message it has ever seen. Just like humans, long-term memory in agents comes in three flavors: semantic memory (facts and knowledge), episodic memory (recall of past experiences or completed tasks), and procedural memory (learned 'how-to' knowledge — internalized prompts/instructions for performing a skill). This isn't a nice-to-have: it's what lets agents learn from past interactions and adapt to new tasks without ever touching the underlying LLM's weights — a form of continual learning implemented entirely at the system level.
Why it exists
Without memory, an agentic system is stateless: in iteration #1 a user mentions their favorite color, and in iteration #2 the agent knows nothing about it — every interaction is a blank slate. Deployed in production, this means it doesn't matter if a user told the agent their name five seconds ago or if the agent helped troubleshoot an issue in the last session — none of it is remembered. Memory exists to make agents context-aware and practically usable across sessions, turning a series of disconnected Q&A exchanges into a coherent, evolving relationship with the user.
Problem it solves
Memory solves the 'goldfish agent' problem — support bots that ask users to repeat their issue every message, coding assistants that forget the architecture decisions made two hours ago, personal assistants that can't recall a stated preference from yesterday. It also solves the continual-learning problem without retraining: instead of fine-tuning a model every time you want it to 'learn' something new about a user or task, you write that information to a memory store and retrieve it at the right moment — cheaper, faster, and reversible (you can simply delete a memory) compared to weight updates.
Intuition
Think about the difference between talking to a stranger versus talking to a close colleague you've worked with for years. The stranger (a memory-less agent) needs every piece of context re-explained from scratch every single conversation. The colleague (a memory-equipped agent) remembers your preferences, your past projects, the jargon you use, and the mistakes you've both learned from — every new conversation builds on that shared history instead of starting over.
Analogy
Short-term memory is like a sticky note on your monitor for today's meeting notes — useful right now, gone tomorrow. Long-term memory is like a filed-away project folder — low value in the moment but retrievable months later. Entity memory is like your phone's contacts list — specific facts pinned to a specific person or thing. Contextual memory is like remembering 'we're in the middle of discussing Q3 budget' so you don't have to be reminded every message. User memory is like a loyalty-program profile — persistent preferences tied to one specific user across every visit.
Technical explanation
The three long-term memory types mirror human cognitive psychology: semantic memory stores decontextualized facts (e.g., 'the user's company uses Python 3.11 and PostgreSQL'); episodic memory stores time-stamped records of specific past events or completed tasks (e.g., 'on March 3rd, the agent successfully debugged a Redis connection timeout for this user'); procedural memory stores internalized how-to knowledge — effectively cached, refined instructions/prompts for performing a recurring skill well (e.g., a refined system prompt for 'how to write this team's preferred commit message style', learned from repeated corrections).
Beyond this long-term taxonomy, a practical agent memory system is typically decomposed into five architectural layers: Short-Term Memory (the current conversation's message history — cleared at session end), Long-Term Memory (persisted across sessions, typically in a vector store or database, holding semantic/episodic/procedural content), Entity Memory (structured facts pinned to specific named entities — a person, a project, a company), Contextual Memory (the current task/conversation state — what's been established so far in this specific multi-step interaction), and User Memory (persistent, user-specific preferences and history that follow one user across every session, like a profile).
Crucially, memory is not a property of the LLM itself — the model has no built-in persistence between API calls. To simulate memory, the system must explicitly manage context: choosing what to keep, what to discard, and what to retrieve before every new model call. This makes memory a system design problem, not a model capability.
Architecture
A production agent memory system typically has: a Short-Term store (in-process list or Redis, holding the active conversation, evicted at session end or via summarization when it grows too large), a Long-Term store (a vector database like Pinecone/Weaviate/pgvector for semantic search over past episodic/semantic memories, often paired with a relational store for structured entity/user facts), a Memory Writer (a background or inline process that decides what's worth persisting from an interaction — not everything should be saved), a Memory Retriever (runs before each LLM call — queries the long-term store for memories relevant to the current turn, via semantic similarity, entity match, or recency), and a Context Assembler (merges retrieved memories + short-term history + system prompt into the final prompt sent to the LLM, respecting context-window limits).
Workflow
- Classify what needs to be remembered: is it a fact (semantic), an event (episodic), a refined skill (procedural), a specific entity's attributes, or a general user preference?
- Choose a storage backend per memory type: short-term = in-memory list/Redis; long-term semantic/episodic = vector store; entity/user = structured DB or key-value store.
- After each agent turn, run a Memory Writer step that decides whether anything from this turn is worth persisting (not every message is memory-worthy — filter aggressively).
- Before each new agent turn, run a Memory Retriever step that pulls relevant memories (via semantic search, entity lookup, or recency) based on the current user input.
- Assemble the final prompt: system prompt + retrieved long-term memories + entity/user memory + short-term conversation history + current input — trimming to fit the context window, prioritizing the most relevant/recent items.
- Periodically summarize/compress short-term memory into long-term memory as conversations grow long, to avoid unbounded context growth.
Example
from dataclasses import dataclass, field from datetime import datetime
@dataclass class Memory: kind: str # 'semantic' | 'episodic' | 'procedural' content: str timestamp: datetime = field(default_factory=datetime.utcnow)
class AgentMemory: def init(self, embed_fn, vector_store, entity_store: dict): self.embed_fn = embed_fn self.vector_store = vector_store # long-term semantic/episodic self.entity_store = entity_store # entity/user memory (key-value) self.short_term: list[str] = [] # current session messages
def write(self, memory: Memory, entity_key: str | None = None):
if entity_key:
self.entity_store.setdefault(entity_key, []).append(memory.content)
else:
vec = self.embed_fn(memory.content)
self.vector_store.upsert(vec, metadata={'kind': memory.kind, 'text': memory.content})
def retrieve(self, query: str, entity_key: str | None = None, k: int = 5) -> list[str]:
results = []
if entity_key and entity_key in self.entity_store:
results += self.entity_store[entity_key]
vec = self.embed_fn(query)
results += [m['text'] for m in self.vector_store.search(vec, top_k=k)]
return results
def assemble_prompt(self, system_prompt: str, query: str, entity_key: str | None = None) -> str:
memories = self.retrieve(query, entity_key)
memory_block = '\n'.join(f'- {m}' for m in memories)
return f"{system_prompt}\n\nRelevant memory:\n{memory_block}\n\nUser: {query}"
Real-world usage
ChatGPT's 'Memory' feature persists user facts (name, preferences, ongoing projects) across sessions using exactly this entity/user-memory pattern, retrieved and injected into context at the start of new conversations. Character.ai and other companion-chat products rely heavily on episodic memory to make characters feel like they 'remember' past conversations, which is core to their product's perceived quality. Coding assistants like Cursor and GitHub Copilot Workspace use contextual memory (the current task/PR state) plus entity memory (facts about the specific codebase/repo) to avoid re-explaining project structure on every request. Customer support agent platforms (Intercom Fin, Ada) use long-term episodic memory to recall a specific customer's past tickets, avoiding the 'please explain your issue again' experience that erodes user trust.
Trade-offs
Adding memory increases system complexity (a writer, a retriever, a storage backend, a context assembler) and cost (extra retrieval calls, storage costs, and more tokens injected into every prompt) in exchange for continuity that plain stateless agents cannot offer. Over-aggressive memory writing (saving everything) leads to noisy retrieval and context bloat; under-aggressive memory writing loses genuinely useful context. The right level of memory-writing aggressiveness is task-dependent: a one-off Q&A tool needs none, while a long-term personal assistant needs a carefully tuned writer/retriever pipeline.
Visual explanation
Two side-by-side diagrams.
Left: 'Agentic system WITHOUT memory' — Iteration 1: [User: 'My favorite color is blue'] → Agent responds. Iteration 2: [User: 'What's my favorite color?'] → Agent: 'I don't know' (no connection between iterations, each is an isolated box).
Right: 'Agentic system WITH memory' — Iteration 1: [User: 'My favorite color is blue'] → Agent responds AND writes {user_preference: favorite_color=blue} to a Memory Store. Iteration 2: [User: 'What's my favorite color?'] → Agent reads Memory Store → 'Blue!' — an arrow connects the memory store across both iterations, showing persistence.
Advantages
- —
Enables continual learning and personalization without ever touching model weights
- —
Makes agents context-aware across sessions instead of stateless one-shot responders
- —
Different memory types (semantic/episodic/procedural, short/long-term/entity/contextual/user) let you store exactly the right kind of information for the right retrieval pattern
- —
Memories are inspectable, editable, and deletable — far more controllable than 'what a fine-tuned model learned'
Disadvantages
- —
Adds real system complexity: a writer, a retriever, a storage backend, and a context assembler all need to be built and maintained
- —
Costs more: extra storage, extra retrieval calls, and more tokens injected into every prompt
- —
Over-aggressive memory writing creates noisy retrieval and context bloat; under-aggressive writing loses useful history
- —
Stale or incorrect memories can actively mislead an agent if not periodically reviewed/pruned
Common mistakes
- —
Treating memory as a single undifferentiated blob instead of separating short-term, long-term, entity, contextual, and user memory — each needs different storage and retrieval strategies
- —
Writing every single message to long-term memory instead of filtering for what's actually worth persisting, leading to noisy, low-precision retrieval later
- —
Never pruning or expiring stale memories, so outdated facts (an old job title, a resolved issue) keep polluting context indefinitely
- —
Forgetting that memory is a system design problem, not a model feature — expecting an LLM API call alone to 'just remember' without an explicit write/retrieve pipeline
- —
Not budgeting context-window space for retrieved memories, causing them to get truncated or crowd out the actual current user message
📂 Subtopics
In-Context Memory: What Fits in the Context Window
The simplest form of agent memory is just what's currently sitting in the prompt — conversation history, retrieved documents, tool outputs. It's fast and free of infrastructure, but it's bounded by the context window and disappears the moment the session ends.
~12 min
External Memory: Vector Stores & Databases
External memory persists information outside the context window — in vector stores or databases — so an agent can recall relevant facts from far earlier, or from entirely separate past sessions, without needing everything to fit in one prompt.
~15 min
Episodic vs Semantic vs Procedural Memory in Agents
Long-term agent memory, mirroring human memory, splits into three distinct flavors: semantic (facts), episodic (past experiences), and procedural (learned how-to knowledge) — each stored and used differently.
~15 min
Memory Management Strategies: When to Store, Retrieve, and Forget
Memory isn't a property of the model — it's a system design problem. Deciding explicitly what to write, what to read back in, and when to let something be forgotten is what actually makes agent memory work in practice.
~15 min