How it works
Instead of comparing exact text, semantic cache converts the question into an embedding (vector) and searches the cache for answers to questions with nearby vectors. If similarity exceeds a threshold, the cached answer is returned without calling the model. This avoids repeated calls for equivalent questions.
Practical example
A user asks "how to reset my password" and another asks "how to change my password". The two sentences are different, but the meaning is the same, so the second one gets the cached answer from the first, saving a model call.
What it's for
It reduces cost and latency in AI applications, especially in support and search, where questions repeat with word variations. It is one of the most effective optimizations in production LLM systems.
Risks
If the similarity threshold is too low, the cache can return an inadequate response. The threshold needs to be calibrated and the cache invalidated when information changes.