intermediate~2h

Semantic Caching

Two users ask "what's your refund policy?" and "how do refunds work?" — semantically identical, lexically different. A normal cache misses both. This module doesn't.

Learning objectives

Beginner: A semantic cache in front of a single, high-traffic FAQ endpoint, with a conservative threshold and manual monitoring of hit quality.
Intermediate: Redis-backed semantic cache with a TTL matching how quickly the underlying answers actually change (e.g. daily for promotions, indefinite for policy text).
Advanced: Combine with observability (Module 17) to track cache hit rate and false-positive rate over time, adjusting the threshold based on real production query patterns rather than a value chosen once at launch.

◆ The problem

A traditional cache keys on exact string match — "what's your refund policy?" and "how do refunds work?" are different cache keys entirely, even though a correct answer to one is a correct answer to the other. Every rephrased-but-identical question re-triggers a full (possibly RAG-augmented) LLM call: the latency and cost of Module 09's whole pipeline, paid again for a question you've effectively already answered.

A semantic cache keys on meaning instead of exact text: embed the incoming question, and if a sufficiently similar question (above a similarity threshold) exists in the cache, return its stored answer without calling the model at all.

This directly reuses the vector-search machinery from Module 08: the "cache" is itself a small vector store. Cache lookup is a similarity search; a cache write is inserting a new embedded (question, answer) pair.

@Service
public class SemanticCacheService {

    private final VectorStore cacheStore; // a dedicated collection, separate from RAG's own store
    private static final double SIMILARITY_THRESHOLD = 0.85; // sweet spot is ~0.7-0.9 — see pitfall below

    public Optional<String> lookup(String question) {
        return cacheStore.similaritySearch(
                SearchRequest.builder().query(question).topK(1).similarityThreshold(SIMILARITY_THRESHOLD).build())
            .stream()
            .findFirst()
            .map(doc -> (String) doc.getMetadata().get("answer"));
    }

    public void store(String question, String answer) {
        cacheStore.add(List.of(new Document(question, Map.of("answer", answer))));
    }
}

💻 Code example

@Service
public class SemanticCacheService {

    private final VectorStore cacheStore; // a dedicated collection, separate from RAG's own store
    private static final double SIMILARITY_THRESHOLD = 0.85; // sweet spot is ~0.7-0.9 — see pitfall below

    public Optional<String> lookup(String question) {
        return cacheStore.similaritySearch(
                SearchRequest.builder().query(question).topK(1).similarityThreshold(SIMILARITY_THRESHOLD).build())
            .stream()
            .findFirst()
            .map(doc -> (String) doc.getMetadata().get("answer"));
    }

    public void store(String question, String answer) {
        cacheStore.add(List.of(new Document(question, Map.of("answer", answer))));
    }
}

▲ Pitfall

This is the entire design risk of semantic caching: set the threshold too low, and genuinely different questions ("what's your refund policy?" vs. "what's your exchange policy?") can be similar enough to incorrectly return the wrong cached answer — a false cache hit is worse than a cache miss, because it's silently, confidently wrong. Set it too high (approaching 1.0), and matching becomes too strict — you lose most of the benefit, rarely matching anything except near-exact rephrasings. The practical sweet spot most teams land on is 0.7–0.9, tuned against real query logs rather than guessed at — not the near-1.0 value intuition suggests "safe" would be.

◆ Under the hood — Redis vs. a dedicated vector store for caching

Redis, with its vector search module, is a common choice specifically for semantic caching (as distinct from the main RAG knowledge store) because cache entries are typically short-lived, high-throughput, and benefit from Redis's existing TTL/eviction semantics — a cached answer to a question about "today's promotions" should naturally expire, which a general-purpose RAG document store isn't optimized to do.

✓ Quick recap

What does a semantic cache key on, instead of exact string match? Embedding similarity — a rephrased but semantically identical question can still hit the cache. Why is a false cache hit worse than a cache miss? It returns a wrong answer confidently and silently, rather than simply costing an extra (correct) model call.

Want a visual for this concept?

Generate a diagram tailored to “Semantic Caching” — the AI picks whichever visual (flowchart, comparison, sequence, etc.) best fits.

Practice quiz

Next Step

Continue to ETL Pipeline for RAG Ingestion →← Back to all Spring AI chapters