Redis introduces langcache: a semantic cache for llms that cuts costs by up to 90%

Redis has launched LangCache, a managed semantic caching service designed to reduce LLM API costs by up to 90% and accelerate cache hits by up to 15 times. The service operates between the application and the model, matching incoming prompts by meaning rather than exact text, thereby avoiding redundant model calls. LangCache is currently in public preview on Redis Cloud, accessible via REST API with Python and JavaScript SDKs. The system stores generated responses and uses Redis’s vector database to manage cache behavior through similarity thresholds, TTLs, and eviction policies. In a demo, LangCache returned a response in 0.37 seconds, compared to 2.232 seconds for direct inference. Redis claims that cache hits eliminate input and output token costs, with estimated monthly savings calculated as (Monthly output token costs) x (Cache hit rate). A case study from Mangoes.ai reported a 70% reduction in LLM spend with 4x faster responses. Redis emphasizes the importance of well-tuned thresholds and data isolation, ensuring that cache hits are both accurate and efficient. The service is available at redis.io/langcache, with additional resources on social media and a savings calculator for annual estimates.