Benefits of Semantic Caching for Repeated LLM Questions

Your support bot has now answered "How do I reset my password?" in a dozen phrasings and billed you for every one. Semantic caching for an LLM app fixes that: it turns each incoming question into an embedding (a list of numbers that captures meaning), looks for a stored question that means the same thing, and returns the saved answer instead of calling the model again.

How semantic caching works for repeated LLM questions

A normal cache needs identical text to get a hit. A semantic cache matches by embedding similarity above a threshold, so "What are Product A's features?" and "Tell me about Product A's capabilities" can share one cached answer.

Here is the whole idea in code, using RedisVL's SemanticCache. You need a running Redis Stack or Redis Cloud instance and pip install redisvl sentence-transformers.

from redisvl.extensions.cache.llm import SemanticCache
from redisvl.utils.vectorize import HFTextVectorizer

cache = SemanticCache(
    name="faq_cache",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,  # Redis COSINE distance [0-2], lower is stricter
    vectorizer=HFTextVectorizer("redis/langcache-embed-v1"),
)
cache.set_ttl(86400)  # entries expire after one day

stats = {"hits": 0, "misses": 0}


def answer(question: str) -> str:
    if results := cache.check(prompt=question):
        stats["hits"] += 1
        return results[0]["response"]
    stats["misses"] += 1
    response = call_llm(question)  # your existing model call
    cache.store(prompt=question, response=response)
    return response


def hit_rate() -> float:
    total = stats["hits"] + stats["misses"]
    return stats["hits"] / total if total else 0.0

The wrapper checks the cache first, calls your model only on a miss, stores the new answer, and counts hits and misses. call_llm stands for whatever function you already use to ask the model.

Five benefits of LLM response caching

1. You pay per unique question, not per question

Redis documents the savings estimate as monthly output-token cost times cache hit rate. Its worked example is $200 of monthly spend, 60% of it output tokens ($120), and a 50% hit rate: about $60 a month back. Redis says input-token costs are typically offset by embedding and storage costs, so count output tokens only. Rerun the formula on your own bill to see how far you can reduce LLM API cost.

2. Repeat questions come back faster

In RedisVL's own caching guide, one OpenAI call took 1.79 seconds, while the cached answer averaged 0.049 seconds over ten repeats, a 97.26% time saving. That is one short-answer demo, so treat it as an illustration and time your own median latency before and after.

3. Hit rate gives the project a number to track

The hit_rate() function above is the metric. Report it next to spend and forecast with it: hit rate times output-token spend is your expected saving.

For a reference point, an academic study of a Redis-backed embedding similarity cache reported hit rates of 61.6% to 68.8% across its query categories and positive-hit accuracy above 97% (how often a served hit was correct). That is one benchmark, not a promise. Your first week of logs sets your real baseline.

4. The risk becomes a dial you can set

Semantic caching gives up the guarantee that a hit is correct, because a close-enough question can match a different one. The similarity threshold is the control. LangCache defaults to 0.85 with a recommended starting range of 0.8 to 0.9, and Redis advises starting tight, then loosening gradually while you watch hit rate and spot-check matches.

The right setting depends on what a wrong answer costs against an extra model call: an FAQ bot can run looser than one answering account-specific financial questions. That is a decision the business can sign off on. In RedisVL the knob is cache.set_threshold(0.1), in distance units where lower is stricter, the opposite direction from LangCache's similarity score.

5. It is a small, reversible change

The cache wraps your existing model call, so prompts and model stay untouched. If a bad answer gets stored, cache.clear() invalidates everything, and set_ttl() lets entries expire on their own. Rolling back means bypassing the wrapper.

When semantic caching is not worth it

• Few repeats. Every miss still pays for an embedding and a lookup, so a workload where questions rarely repeat gets the cost without the benefit. Sample a week of real questions and count near-duplicates before you build anything.
• Personalized answers. RedisVL's guide stores two nearly identical prompts about the phone number on an account, with different answers, and keeps them apart with a user_id tag filter. If the answer depends on who is asking, scope the cache like this or skip it.
• Fast-changing data. Prices, stock levels and incident status go stale. Set a TTL shorter than the data's shelf life, or leave those questions uncached.

from redisvl.extensions.cache.llm import SemanticCache
from redisvl.query.filter import Tag

private_cache = SemanticCache(
    name="private_cache",
    redis_url="redis://localhost:6379",
    filterable_fields=[{"name": "user_id", "type": "tag"}],
)

private_cache.store(
    prompt="What is the phone number linked to my account?",
    response="The number on file is 123-555-0000",
    filters={"user_id": "abc"},
)

hits = private_cache.check(
    prompt="What is the phone number linked to my account?",
    filter_expression=Tag("user_id") == "abc",
)

check only returns entries matching the user_id filter. Without it, a near-identical question from another user could match the first user's stored answer.

For anyone running an LLM in production, semantic caching is one of the few cost levers that arrives with its own dashboard number, so ship it behind a flag, watch the hit rate for a week, and let the data decide whether it stays.