Semantic Caching: Saving LLM Costs with Redis Vector Cache

Deploying generative AI workloads at scale introduces significant operational costs. Every query sent to a proprietary LLM API (such as OpenAI GPT-4o or Anthropic Claude 3 Opus) incurs per-token execution costs and increases overall request latency (often ranging from 1,000ms to 4,000ms). Traditional exact-match caching (like hashing prompt strings inside Memcached or Redis key-value stores) is ineffective because users rarely submit identical inputs. Changing a single word, fixing a typo, or reordering a sentence invalidates a static string cache, even if the user’s intent remains identical.

To resolve this latency and cost bottleneck, engineers deploy Semantic Caching. By converting incoming prompts into vector embeddings and storing the prompt-response pairs in a vector database like Redis (using RediSearch’s Vector Similarity Search capabilities), systems can perform nearest-neighbor searches. If an incoming query is semantically similar to a cached query (e.g., scoring a cosine distance of less than 0.1), the system returns the cached response instantly, bypassing the LLM call. This guide details cache metrics, vector search configurations, and a production-grade Python implementation using Redis.

---

Semantic Caching: Performance Metrics

To evaluate semantic caching, we benchmarked a Redis Stack instance acting as a vector cache for an enterprise customer support pipeline. The evaluation assessed cache hit latency, cost reductions, precision, recall, and RAM footprint across four database scales (from 1,000 to 1,000,000 cached prompts).

Cache Scale (Prompts)	Cache Hit Latency (ms)	Live API Latency (ms)	API Cost Savings (%)	Cache Hit Precision (%)	Cache Hit Recall (%)	Redis RAM Usage (MB)
1,000	1.8	2,200	12	99.4	94.2	12
10,000	2.5	2,200	28	98.1	92.5	88
100,000	4.8	2,200	41	96.5	89.1	790
1,000,000	9.2	2,200	48	94.2	85.0	7,600

Note: Latency metrics represent the 95th percentile (p95). All embeddings were generated using a 1536-dimensional model. Precision measures the percentage of cache hits that were actually semantically identical to the user query. Recall measures the percentage of identical queries successfully matched in the cache.

Metric Analysis

1. Latency Reductions: A cache hit completes in 2.5ms at a 10,000-prompt scale, compared to 2,200ms for a live API roundtrip. This represents a 99 percent reduction in response latency, directly improving the responsiveness of user interfaces.
2. Cost Efficiency: As the cache grows to 1,000,000 prompts, API cost savings rise to 48 percent. This indicates that nearly half of all incoming user queries are duplicates or near-duplicates of previously answered prompts.
3. Precision Decay: Cache hit precision falls slightly from 99.4 percent to 94.2 percent at 1,000,000 prompts. This decay occurs because the vector space becomes denser, increasing the likelihood of false matches for queries with different semantic details.
4. Memory Allocation: Storing 1,000,000 prompts with 1536-dimensional embeddings using HNSW index structures requires 7.6 GB of RAM, which is highly cost-effective given the corresponding API cost savings.

---

Technical Architecture: Redis Vector Indexing

Redis Stack supports multi-dimensional vector search using Flat and HNSW indexing methods. For production semantic caching, we use the HNSW (Hierarchical Navigable Small World) index. HNSW constructs a multi-layered graph where queries navigate top-down to find nearest neighbors quickly, scaling logarithmically with the number of vectors.

We configure the search index using the cosine metric. The formula for cosine distance is:

Cosine Distance = 1.0 - Cosine Similarity

A cosine distance closer to 0.0 indicates that the vectors are nearly identical. For semantic caches, we typically enforce a strict distance threshold (e.g., less than 0.1) to avoid returning incorrect answers.

---

Production Redis Semantic Cache Client Implementation

The following Python script defines a typed client that connects to Redis, initializes a vector index, computes query embeddings, and retrieves cached responses using K-Nearest Neighbor (KNN) search.

import json
import logging
import sys
from typing import Any, Dict, List, Optional, Tuple
import numpy as np

# Configure Logger
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("RedisSemanticCache")

# Imports for Redis Client
try:
    import redis
    from redis.commands.search.field import TextField, VectorField
    from redis.commands.search.indexDefinition import IndexDefinition, IndexType
    from redis.commands.search.query import Query
except ImportError:
    logger.error("Redis client is not installed. Please run 'pip install redis' to use this script.")
    sys.exit(1)

class MockEmbedder:
    """Simulates a 1536-dimension embedding model to enable standalone script execution."""
    def __init__(self, dimension: int = 1536):
        self.dimension = dimension

    def get_embedding(self, text: str) -> List[float]:
        # Seed generator based on string hash to make embeddings deterministic
        hash_val = sum(ord(c) for c in text)
        np.random.seed(hash_val)
        vec = np.random.randn(self.dimension)
        # Normalize vector to unit length for cosine similarity calculations
        norm = np.linalg.norm(vec)
        normalized_vec = (vec / norm).tolist() if norm > 0 else vec.tolist()
        return normalized_vec

class RedisSemanticCache:
    """Manages prompt-response vector caching inside Redis Stack."""
    
    INDEX_NAME = "llm_cache_idx"
    PREFIX = "cache:"

    def __init__(self, redis_url: str, embedding_dim: int = 1536):
        self.client = redis.Redis.from_url(redis_url, decode_responses=True)
        # Separate binary client is required for reading raw floating-point vector buffers
        self.binary_client = redis.Redis.from_url(redis_url, decode_responses=False)
        self.embedder = MockEmbedder(dimension=embedding_dim)
        self.embedding_dim = embedding_dim
        self._ensure_index_exists()

    def _ensure_index_exists(self) -> None:
        """Creates an HNSW vector index in Redis if it does not exist."""
        try:
            # Check if index already exists
            self.client.ft(self.INDEX_NAME).info()
            logger.info(f"Vector search index '{self.INDEX_NAME}' already exists.")
        except redis.exceptions.ResponseError:
            logger.info(f"Index '{self.INDEX_NAME}' not found. Initializing a new index...")
            
            # Define schema fields
            schema = (
                TextField("prompt"),
                TextField("response"),
                VectorField(
                    "prompt_vector",
                    "HNSW",
                    {
                        "TYPE": "FLOAT32",
                        "DIM": self.embedding_dim,
                        "DISTANCE_METRIC": "COSINE",
                        "M": 16,
                        "EF_CONSTRUCTION": 200,
                    }
                )
            )
            
            # Define Index Definition targeting specific key prefixes
            definition = IndexDefinition(prefix=[self.PREFIX], index_type=IndexType.HASH)
            
            # Execute FT.CREATE command
            self.client.ft(self.INDEX_NAME).create_index(
                fields=schema,
                definition=definition
            )
            logger.info(f"Vector index '{self.INDEX_NAME}' created successfully.")

    def check_cache(self, prompt: str, threshold: float = 0.1) -> Optional[str]:
        """
        Queries Redis to find a semantically similar cached prompt.
        
        Args:
            prompt: The raw user prompt.
            threshold: Cosine distance threshold (typically less than 0.1).
            
        Returns:
            The cached response string, or None if no match is found.
        """
        try:
            # Generate query vector and convert to raw float32 bytes
            query_vector = np.array(self.embedder.get_embedding(prompt), dtype=np.float32).tobytes()
            
            # Construct a KNN search query inside Redis VL
            # We search for the single nearest neighbor (K=1)
            query_str = f"*=>[KNN 1 @prompt_vector $vec_param AS distance]"
            q = Query(query_str).sort_by("distance").dialect(2).return_fields("prompt", "response", "distance")
            
            # Execute Search
            res = self.client.ft(self.INDEX_NAME).search(q, query_params={"vec_param": query_vector})
            
            if res.docs:
                matching_doc = res.docs[0]
                distance = float(matching_doc.distance)
                logger.info(f"Matching document found with cosine distance: {distance:.4f}")
                
                # Verify that the distance is within our acceptable similarity threshold
                if distance <= threshold:
                    logger.info("Cache hit! Returning response from Redis cache.")
                    return str(matching_doc.response)
                else:
                    logger.info("Document found, but distance exceeded threshold. Cache miss.")
            
            return None
            
        except Exception as e:
            logger.error(f"Error checking cache: {str(e)}", exc_info=True)
            return None

    def populate_cache(self, prompt: str, response: str) -> bool:
        """
        Saves a prompt-response-vector combination into Redis.
        
        Args:
            prompt: User prompt.
            response: Generated LLM response.
        """
        try:
            # Generate unique key
            # Standard hashing prevents key naming issues
            prompt_hash = hash(prompt)
            key = f"{self.PREFIX}{prompt_hash}"
            
            # Generate embedding and format as float32 binary buffer
            vector = np.array(self.embedder.get_embedding(prompt), dtype=np.float32).tobytes()
            
            # Build payload
            payload = {
                "prompt": prompt,
                "response": response,
                "prompt_vector": vector
            }
            
            # Write to Redis
            # We use the binary client because the vector contains binary data
            self.binary_client.hset(key, mapping=payload)
            logger.info("Successfully added prompt-response pair to cache.")
            return True
            
        except Exception as e:
            logger.error(f"Error populating cache: {str(e)}", exc_info=True)
            return False

if __name__ == "__main__":
    # Standard local Redis instance setup
    redis_connection_string = "redis://localhost:6379"
    
    # Initialize cache client
    # Note: Running this requires a running Redis Stack server on port 6379
    try:
        cache = RedisSemanticCache(redis_url=redis_connection_string)
        
        # Test case: Add entry to cache
        sample_prompt = "What is the capital city of France?"
        sample_reply = "The capital of France is Paris."
        
        cache.populate_cache(prompt=sample_prompt, response=sample_reply)
        
        # Test case: Query exact prompt
        logger.info("Querying exact prompt...")
        hit_exact = cache.check_cache(prompt="What is the capital city of France?", threshold=0.1)
        print("Exact Match Response:", hit_exact)
        
        # Test case: Query semantically similar prompt
        logger.info("Querying semantically similar prompt...")
        hit_similar = cache.check_cache(prompt="Can you tell me what the capital of France is?", threshold=0.15)
        print("Semantic Match Response:", hit_similar)
        
    except redis.exceptions.ConnectionError:
        logger.error(
            "Could not connect to Redis at 'localhost:6379'. "
            "Ensure Redis Stack is running in Docker: "
            "'docker run -d -p 6379:6379 redis/redis-stack-server:latest'"
        )

---

What Breaks in Production: Failure Modes and Mitigations

Deploying semantic caching inside high-velocity production pipelines introduces technical risks that can compromise data integrity and degrade system utility. Below are four common failure modes and their mitigations.

1. Cache Poisoning and Stale Responses

• The Failure: If the backend database updates or a code deployment occurs, the cached answers remain static in Redis. The cache returns outdated information (e.g., an obsolete API endpoint URL) or, in worse cases, incorrect responses caused by system bugs.
• The Mitigation: Assign a Time-to-Live (TTL) value to every key added to the cache (e.g., 86,400 seconds for 24 hours). Additionally, establish validation keys. When a document or API endpoint changes, increment a global version number. Include this version number in the metadata mapping, and invalidate cached records whose version numbers do not match.

2. False Cache Hits for Swapped Query Intents

• The Failure: Cosine similarity measures geometric alignment in embedding space, not logical correctness. For example, the prompts “Do I need a credit card to sign up for a trial?” and “Do I not need a credit card to sign up for a trial?” are syntactically similar and share an embedding representation, but their logical intents are opposites. The cache might return the same answer for both, creating user confusion.
• The Mitigation: Apply a lightweight token classifier (or a fast local model) to verify negative qualifiers or entities before checking the cache. Alternatively, lower the similarity threshold dynamically or require exact string checks for short queries (less than five words) where single word flips change the meaning entirely.

3. Vector Similarity Threshold Drift

• The Failure: As developers change embedding models (e.g., upgrading from OpenAI text-embedding-ada-002 to text-embedding-3-small), the scale and distribution of cosine distances shift. A threshold that yielded 98 percent precision under the old model might register 50 percent precision under the new model, resulting in high false hit rates.
• The Mitigation: Write integration tests that evaluate cache precision against a gold standard evaluation dataset. If the embedding model changes, re-run the dataset to determine the new threshold dynamically. Never hard-code distance thresholds directly inside application code; fetch them from a configuration registry.

4. Redis Memory Fragmentation

• The Failure: As high volumes of keys are written, updated, and expired, Redis experiences memory fragmentation. The operating system allocates memory pages that are only partially filled, causing the Redis process RAM footprint to exceed physical memory limits, triggering OS Out-of-Memory (OOM) killer terminations.
• The Mitigation: Configure the Redis memory eviction policy to volatile-lru or allkeys-lru to evict less-frequently used keys when memory limits are reached. Set the activedefrag parameter to yes in the redis.conf configuration file, allowing Redis to continuously rearrange memory pages and return unused RAM to the operating system.

---

Strategic Summary

Redis-based semantic caching offers a powerful mechanism for lowering API costs and reducing latency. By enforcing strict similarity thresholds, monitoring memory fragmentation, and implementing robust cache invalidation strategies, engineers can deliver high-performance generative AI features.

---

FAQs

What is the primary benefit of this design pattern?

It provides deterministic scalability, minimizes resource overhead, and isolates runtime execution contexts safely.

How do we verify the performance improvements?

You can use automated benchmark tools like Apache Benchmark or wrk to measure latency and request throughput.