What is semantic caching and how can it reduce LLM costs in Spring AI?

Question

Standard caching matches by exact prompt string. Semantic caching embeds the incoming prompt and searches a vector store for a similar-enough past query (above a similarity threshold), returning its cached answer instead of calling the LLM again — implemented as a custom CallAdvisor, useful for reducing cost on repeated or paraphrased questions.

CrackLab · Accepted Answer

Standard caching matches by exact prompt string. Semantic caching embeds the incoming prompt and searches a vector store for a similar-enough past query (above a similarity threshold), returning its cached answer instead of calling the LLM again — implemented as a custom CallAdvisor, useful for reducing cost on repeated or paraphrased questions.

What is semantic caching and how can it reduce LLM costs in Spring AI?

Next Step