|
This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0! |
Semantic Caching
Semantic caching is a powerful optimization technique that leverages Redis vector search capabilities to cache and retrieve AI chat responses based on the semantic similarity of user queries rather than exact string matching. This enables intelligent response reuse even when users phrase similar questions differently.
Why Semantic Caching?
Traditional caching relies on exact key matches, which fails when users ask semantically equivalent questions with different wording:
-
"What is the capital of France?"
-
"Tell me France’s capital city"
-
"Which city is the capital of France?"
All three queries have the same answer, but traditional caching would treat them as different requests, resulting in redundant LLM API calls. Semantic caching solves this by comparing the meaning of queries using vector embeddings.
Benefits:
-
Reduced API costs: Avoid redundant calls to expensive LLM APIs
-
Lower latency: Return cached responses instantly instead of waiting for model inference
-
Improved scalability: Handle higher query volumes without proportional API cost increases
-
Consistent responses: Return identical answers for semantically similar questions
Auto-configuration
Spring AI provides Spring Boot auto-configuration for the Redis Semantic Cache.
There is no starter for it.
Add the auto-configuration, the semantic cache module, spring-ai-transformers (the auto-configuration requires it on the classpath) and Spring Boot’s Redis starter to your project’s Maven pom.xml file:
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-autoconfigure-vector-store-redis-semantic-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-redis-semantic-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-transformers</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
or to your Gradle build.gradle build file:
dependencies {
implementation 'org.springframework.ai:spring-ai-autoconfigure-vector-store-redis-semantic-cache'
implementation 'org.springframework.ai:spring-ai-redis-semantic-cache'
implementation 'org.springframework.ai:spring-ai-transformers'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
}
The auto-configuration connects to Redis through the JedisConnectionFactory that Spring Boot creates from the spring.data.redis.* properties.
Spring Boot uses Lettuce by default, so set spring.data.redis.client-type=jedis; Jedis comes with spring-ai-redis-semantic-cache.
The auto-configuration provides a default embedding model optimized for semantic caching (redis/langcache-embed-v1).
You can override this by providing your own EmbeddingModel bean.
|
Configuration Properties
Properties starting with spring.ai.vectorstore.redis.semantic-cache.* configure the semantic cache:
| Property | Description | Default Value |
|---|---|---|
|
Enable or disable the semantic cache |
|
|
Similarity threshold for cache hits (0.0-1.0). Higher values require closer semantic matches. |
|
|
Name of the Redis search index for cache entries |
|
|
Key prefix for cached entries in Redis |
|
|
Time-to-live of the responses cached by the auto-configured |
- |
The Redis host, port and credentials are Spring Boot’s spring.data.redis.* properties.
Example configuration in application.yml:
spring:
data:
redis:
host: localhost
port: 6379
client-type: jedis
ai:
vectorstore:
redis:
semantic-cache:
enabled: true
similarity-threshold: 0.85
index-name: my-app-cache
prefix: "my-app:semantic-cache:"
Using the SemanticCacheAdvisor
The SemanticCacheAdvisor integrates seamlessly with Spring AI’s ChatClient advisor pattern.
It automatically caches responses and returns cached results for similar queries.
The advisor requires a SemanticCacheScopeResolver that decides which requests may share cached responses, see Cache Scope:
@Autowired
private SemanticCache semanticCache;
@Autowired
private ChatModel chatModel;
public void example() {
// Create the cache advisor, sharing cached responses across all users
SemanticCacheAdvisor cacheAdvisor = SemanticCacheAdvisor.builder()
.cache(semanticCache)
.scopeResolver(request -> "shared")
.build();
// First query - calls the LLM and caches the response
ChatResponse response1 = ChatClient.builder(chatModel)
.build()
.prompt("What is the capital of France?")
.advisors(cacheAdvisor)
.call()
.chatResponse();
// Similar query - returns cached response (no LLM call)
ChatResponse response2 = ChatClient.builder(chatModel)
.build()
.prompt("Tell me the capital city of France")
.advisors(cacheAdvisor)
.call()
.chatResponse();
// response1 and response2 contain the same cached answer
}
The advisor automatically:
-
Checks the cache for semantically similar queries before calling the LLM
-
Returns cached responses when a match is found above the similarity threshold
-
Caches new responses after successful LLM calls
-
Supports both synchronous and streaming chat operations
The auto-configuration only registers a SemanticCacheAdvisor bean when the application defines a SemanticCacheScopeResolver bean.
|
Cache Scope
A semantic cache returns the response generated for one request to any other semantically similar request. In a multi-user application, another user asking a similar question to a different user may get access to their data.
To prevent this, the SemanticCacheAdvisor requires a SemanticCacheScopeResolver.
The resolver has access to ChatClientRequest and returns the scope of the request, such as a tenant or user identifier.
Cached responses are only shared between requests that resolve to the same scope.
When the resolver returns null or a blank string, the request bypasses the cache: nothing is read from or written to it.
An approach is to pass the scope as an advisor parameter, which ends up in the request context:
@Bean
public SemanticCacheScopeResolver semanticCacheScopeResolver() {
return request -> (String) request.context().get("tenantId");
}
// For each request
ChatResponse response = chatClient.prompt()
.user("What is my account balance?")
.advisors(a -> a.param("tenantId", tenantId))
.call()
.chatResponse();
The scope must come from a trusted source, such as the authenticated principal, and not from a value the end user can choose.
For streaming requests, the resolver may be invoked on a different thread than the one that issued the request, so prefer the request context over thread-bound
state such as SpringSecurity SecurityContextHolder.
|
Return a constant only when all users are meant to share the same cached responses.
Direct Cache Usage
You can also interact with the SemanticCache directly for fine-grained control:
@Autowired
private SemanticCache semanticCache;
// Store a response with a query
semanticCache.set("What is the capital of France?", chatResponse);
// Store with TTL (time-to-live) for automatic expiration
semanticCache.set("What's the weather today?", weatherResponse, Duration.ofHours(1));
// Retrieve a semantically similar response
Optional<ChatResponse> cached = semanticCache.get("Tell me France's capital");
if (cached.isPresent()) {
// Use the cached response
String answer = cached.get().getResult().getOutput().getText();
}
// Clear all cached entries
semanticCache.clear();
Manual Configuration
For more control, you can manually configure the semantic cache components:
@Configuration
public class SemanticCacheConfig {
@Bean
public RedisClient jedisClient() {
return RedisClient.builder().hostAndPort("localhost", 6379).build();
}
@Bean
public SemanticCache semanticCache(RedisClient jedisClient, EmbeddingModel embeddingModel) {
return DefaultSemanticCache.builder()
.jedisClient(jedisClient)
.embeddingModel(embeddingModel)
.distanceThreshold(0.3) // Lower = stricter matching
.indexName("my-semantic-cache")
.prefix("cache:")
.build();
}
@Bean
public SemanticCacheAdvisor semanticCacheAdvisor(SemanticCache cache) {
return SemanticCacheAdvisor.builder()
.cache(cache)
.scopeResolver(request -> (String) request.context().get("tenantId"))
.build();
}
}
Cache Isolation with Namespaces
For multi-tenant applications or when you need separate cache spaces, use different index names to isolate cache entries:
// Create isolated caches for different users or contexts
SemanticCache user1Cache = DefaultSemanticCache.builder()
.jedisClient(jedisClient)
.embeddingModel(embeddingModel)
.indexName("user-1-cache")
.build();
SemanticCache user2Cache = DefaultSemanticCache.builder()
.jedisClient(jedisClient)
.embeddingModel(embeddingModel)
.indexName("user-2-cache")
.build();
// Each user gets their own isolated cache space
SemanticCacheAdvisor user1Advisor = SemanticCacheAdvisor.builder()
.cache(user1Cache)
.scopeResolver(request -> "user-1")
.build();
System Prompt Isolation
The scope is the only isolation boundary of the SemanticCacheAdvisor: requests with different system prompts share cached responses within a scope.
Applications with multiple AI personas or context-dependent behavior can isolate them by including the system prompt in the scope, so that the same user query with different system prompts returns different cached responses.
SemanticCacheAdvisor cacheAdvisor = SemanticCacheAdvisor.builder()
.cache(semanticCache)
.scopeResolver(request -> {
String tenantId = (String) request.context().get("tenantId");
// No tenant, no scope: bypass the cache instead of sharing a "null" scope
return (tenantId != null) ? tenantId + "|" + request.prompt().getSystemMessage().getText() : null;
})
.build();
// Query with technical support persona
ChatResponse technicalResponse = ChatClient.builder(chatModel)
.build()
.prompt()
.system("You are a technical support specialist. Provide detailed technical answers.")
.user("How do I reset my password?")
.advisors(cacheAdvisor)
.call()
.chatResponse();
// Same query with customer service persona - cache MISS (different context)
ChatResponse serviceResponse = ChatClient.builder(chatModel)
.build()
.prompt()
.system("You are a friendly customer service agent. Keep responses brief and helpful.")
.user("How do I reset my password?")
.advisors(cacheAdvisor)
.call()
.chatResponse();
// Same query with technical support persona again - cache HIT
ChatResponse technicalAgain = ChatClient.builder(chatModel)
.build()
.prompt()
.system("You are a technical support specialist. Provide detailed technical answers.")
.user("How do I reset my password?")
.advisors(cacheAdvisor)
.call()
.chatResponse();
// Returns the cached technical response
How it works:
The advisor computes a deterministic hash of the scope and uses it as a metadata filter when storing and retrieving cached responses:
-
Same scope + semantically similar user question → cache hit
-
Different scope → cache miss, regardless of the user question
-
With the scope resolver above, a different system prompt results in a different scope, and thus a cache miss
Context-Aware Cache API
For advanced use cases, you can use the context-aware cache methods directly:
// Store with explicit context hash
String contextHash = "technical-support-context";
semanticCache.set("How do I reset my password?", response, contextHash);
// Retrieve with context filtering
Optional<ChatResponse> cached = semanticCache.get("How do I reset my password?", contextHash);
// Different context hash returns empty (no match)
Optional<ChatResponse> otherContext = semanticCache.get("How do I reset my password?", "billing-context");
Tuning the Similarity Threshold
The similarity threshold determines how closely a query must match a cached entry to be considered a hit. The threshold is expressed as a value between 0.0 and 1.0:
-
Higher threshold (e.g., 0.95): Requires very close semantic matches. Reduces false positives but may miss valid cache hits.
-
Lower threshold (e.g., 0.70): Allows broader semantic matches. Increases cache hit rate but may return less relevant cached responses.
// Strict matching - only very similar queries hit the cache
SemanticCache strictCache = DefaultSemanticCache.builder()
.jedisClient(jedisClient)
.embeddingModel(embeddingModel)
.distanceThreshold(0.2) // Strict (distance-based, lower = stricter)
.build();
// Lenient matching - broader semantic similarity accepted
SemanticCache lenientCache = DefaultSemanticCache.builder()
.jedisClient(jedisClient)
.embeddingModel(embeddingModel)
.distanceThreshold(0.5) // Lenient
.build();
| Start with a higher threshold (stricter matching) and gradually lower it based on your application’s tolerance for semantic variation. |
TTL and Cache Expiration
Cached responses can be configured with a time-to-live (TTL) for automatic expiration. This is essential for time-sensitive data:
// Cache weather data for 1 hour
semanticCache.set("What's the weather in New York?", weatherResponse, Duration.ofHours(1));
// Cache general knowledge indefinitely (no TTL)
semanticCache.set("What is photosynthesis?", scienceResponse);
// Redis automatically removes expired entries
The SemanticCacheAdvisor stores cached responses without TTL by default, so they never expire.
Configure a TTL on the advisor, or with the spring.ai.vectorstore.redis.semantic-cache.ttl property for the auto-configured advisor, to expire them.
This also limits how long a response generated for one request remains available to other requests of the same scope:
SemanticCacheAdvisor cacheAdvisor = SemanticCacheAdvisor.builder()
.cache(semanticCache)
.scopeResolver(request -> (String) request.context().get("tenantId"))
.ttl(Duration.ofHours(1))
.build();
A TTL requires the DefaultSemanticCache to be backed by a RedisVectorStore, otherwise storing a response fails with an UnsupportedOperationException instead of storing an entry that never expires.
|
How It Works
The semantic cache operates using the following flow:
-
Query embedding: When a query arrives, it is converted to a vector embedding using the configured
EmbeddingModel -
Vector search: With a context hash, which the advisor always passes as the hash of the scope, Redis runs a KNN search for the 5 nearest entries with the same
context_hashand keeps those within the threshold. Without a context hash, when you call theSemanticCachedirectly, it runs a range query (VECTOR_RANGE) that returns the cached entries within the similarity threshold. -
Cache hit: If a semantically similar query is found, the cached
ChatResponseis returned immediately -
Cache miss: If no match is found, the query proceeds to the LLM, and the response is cached for future use
The implementation leverages Redis’s efficient vector indexing (HNSW algorithm) for fast similarity searches, even with large cache sizes.