Memory Management Strategies: When to Store, Retrieve, and Forget

A key reframing worth internalizing: memory is not a property of the model itself. It is a system design problem. An LLM doesn't have memory baked into its weights that updates as it's used — to simulate memory, the system around the model has to explicitly manage three decisions: what to keep, what to discard, and what to retrieve before each new model call.

The 'what to store' decision matters because storing everything is both expensive and counterproductive — a memory system that indiscriminately writes every single message or observation ends up flooding retrieval with noise, making it harder to surface the genuinely relevant memory later. A better default is to store selectively: durable facts (semantic memory), meaningfully distinct episodes worth recalling later (not every routine interaction), and procedural refinements only when there's actual evidence something worked better than before.

The 'what to retrieve' decision is the flip side — even with good storage discipline, pulling back too much irrelevant memory into the context window at query time wastes tokens and can actively confuse the model with irrelevant history. The standard approach is similarity-based retrieval (pull back only the top-k most relevant memories to the CURRENT query) combined with relevance filtering (discard retrieved memories below some minimum relevance score, rather than always including a fixed top-k regardless of how weak the match actually is).

The 'when to forget' decision is the one systems most often skip entirely, to their own detriment. Memory that's stale (a user preference that's since changed), superseded (an old procedure replaced by a refined one), or simply low-value (a one-off episode that never turned out to matter again) should actively be pruned or down-weighted, not left to accumulate forever. Strategies here include time-based decay (older memories matter less unless reinforced), explicit overwriting (a new fact replaces an old one under the same key rather than both coexisting), and periodic consolidation (summarizing many small episodic memories into fewer, denser semantic facts once enough of them accumulate) — all of which keep the memory store useful and fast rather than becoming an ever-growing, increasingly noisy archive.

from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class Memory:
    text: str
    created_at: datetime
    relevance_score: float = 1.0

class ManagedMemoryStore:
    def __init__(self, decay_half_life_days: float = 30, min_retrieval_score: float = 0.5):
        self.memories: list[Memory] = []
        self.decay_half_life_days = decay_half_life_days
        self.min_retrieval_score = min_retrieval_score

    def store(self, text: str, is_significant: bool) -> None:
        # "What to store": skip routine, low-value events entirely
        if is_significant:
            self.memories.append(Memory(text, datetime.now()))

    def _decayed_score(self, memory: Memory, base_similarity: float) -> float:
        age_days = (datetime.now() - memory.created_at).days
        decay = 0.5 ** (age_days / self.decay_half_life_days)  # "when to forget"
        return base_similarity * decay

    def retrieve(self, similarities: list[tuple[Memory, float]]) -> list[str]:
        # "What to retrieve": apply decay, then filter below a relevance floor
        scored = [(m, self._decayed_score(m, sim)) for m, sim in similarities]
        return [m.text for m, score in scored if score >= self.min_retrieval_score]