|
This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0! |
Responses and Streaming
ChatClient Responses
The ChatClient API offers several ways to format the response from the AI Model using the fluent API.
Returning a ChatResponse
The response from the AI model is a rich structure defined by the type ChatResponse.
It includes metadata about how the response was generated and can also contain multiple responses, known as Generations, each with its own metadata.
The metadata includes the number of tokens (each token is approximately 3/4 of a word) used to create the response.
This information is important because hosted AI models charge based on the number of tokens used per request.
An example to return the ChatResponse object that contains the metadata is shown below by invoking chatResponse() after the call() method.
ChatResponse chatResponse = chatClient.prompt()
.user("Tell me a joke")
.call()
.chatResponse();
Finish Reason Categories
Each Generation reports why the model stopped, under the name its provider uses: stop, end_turn, STOP, content_filter, refusal, SAFETY and so on.
getFinishReason() returns that raw name.
getFinishReasonCategory() maps it onto a FinishReasonCategory that is the same for every provider:
| Category | Meaning |
|---|---|
|
The model finished its answer. |
|
A length or token limit cut the answer short. |
|
The model asked for a tool to be called. |
|
A safety filter stopped the answer, or the model refused to answer. |
|
A reason with no meaning that providers share, or one that Spring AI does not know. |
|
The provider gave no reason. |
The category is read next to the raw finish reason:
ChatResponse chatResponse = chatClient.prompt()
.user("Tell me a joke")
.call()
.chatResponse();
ChatGenerationMetadata metadata = chatResponse.getResult().getMetadata();
if (metadata.getFinishReasonCategory() == FinishReasonCategory.FILTERED) {
// A safety filter stopped the answer, or the model refused to answer.
String reason = metadata.getFinishReason();
}
ChatResponse.hasFinishReasonCategory(…) tells whether any generation of the response finished in a category.
Each chat model classifies its own finish reasons, as it knows what the names of its provider mean and sees the whole provider response. This also covers a stop that the finish reason alone does not tell:
-
OpenAI reports a refusal of the model in a field of its own, with an ordinary
stop. The generation isFILTERED, and the refusal is in therefusalmetadata of the message. -
Gemini and Ollama report an ordinary stop when the model asks for a tool. The generation is
TOOL_CALL.
A custom ChatModel sets the category with ChatGenerationMetadata.builder().finishReasonCategory(…).
For a generation that carries no category, it is derived from the raw finish reason by its common names, such as stop, length and content_filter.
On a stream, each chunk is classified by what that chunk holds. A stop that the provider spreads over several chunks, such as a refusal of OpenAI or a tool call of Gemini that is followed by a chunk with the finish reason, is therefore not the category of the last chunk, nor of the aggregated response.
Returning an Entity
You often want to return an entity class that is mapped from the returned String.
The entity() method provides this functionality.
For example, given the Java record:
record ActorFilms(String actor, List<String> movies) {}
You can easily map the AI model’s output to this record using the entity() method, as shown below:
ActorFilms actorFilms = chatClient.prompt()
.user("Generate the filmography for a random actor.")
.call()
.entity(ActorFilms.class);
There is also an overloaded entity method with the signature entity(ParameterizedTypeReference<T> type) that lets you specify types such as generic Lists:
List<ActorFilms> actorFilms = chatClient.prompt()
.user("Generate the filmography of 5 movies for Tom Hanks and Bill Murray.")
.call()
.entity(new ParameterizedTypeReference<List<ActorFilms>>() {});
Reliability Switches: EntityParamSpec
All entity() overloads accept an optional Consumer<EntityParamSpec> that enables two independent, composable behaviors:
-
validateSchema()— validate the JSON response against the entity schema and automatically retry with error feedback on failure. -
useProviderStructuredOutput()— send the schema to the provider as an API-level constraint instead of prompt text.
ActorFilms actorFilms = chatClient.prompt()
.user("Generate the filmography for a random actor.")
.call()
.entity(ActorFilms.class, spec -> spec
.useProviderStructuredOutput()
.validateSchema());
For full details on these switches, supported providers, limitations, and the lower-level converter API, see the Structured Output reference.
When a Safety Filter Stops the Response
A response that a safety filter stopped, or in which the model refused to answer, is usually empty, cut short, or a refusal in plain words.
It is not the entity that was asked for, and parsing it would fail with an error that does not tell why.
When the generation that entity() converts finished in the FILTERED category, and its text is empty or cannot be converted, entity() throws a ContentFilteredException:
try {
ActorFilms actorFilms = chatClient.prompt()
.user("Generate the filmography for a random actor.")
.call()
.entity(ActorFilms.class);
}
catch (ContentFilteredException ex) {
String reason = ex.getFinishReason(); // as the provider named it, for example "content_filter"
String text = ex.getText(); // what the provider returned, possibly empty
}
The parsing error, if there was one, is the cause of the exception. A filtered response that converts all the same is returned as an entity.
The refusal of a guardrail advisor is reported the same way, with a GuardrailRefusalException, which is a ContentFilteredException (see Advisors).
responseEntity() does not throw for a filtered response.
It returns the response with a null entity, and the finish reason is read from the response.
content(), chatResponse() and streaming return the response as the provider left it.
Streaming Responses
The stream() method lets you get an asynchronous response as shown below:
Flux<String> output = chatClient.prompt()
.user("Tell me a joke")
.stream()
.content();
You can also stream the ChatResponse using the method Flux<ChatResponse> chatResponse().
The stream() method has no entity() method.
To get an entity from a stream, aggregate the response and convert it with a Structured Output Converter, as shown below.
This also demonstrates the use of parameters in the fluent API that will be discussed in more detail in a later section of the documentation.
var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<List<ActorFilms>>() {});
Flux<String> flux = chatClient.prompt()
.user(u -> u.text("""
Generate the filmography for a random actor.
{format}
""")
.param("format", converter.getFormat()))
.stream()
.content();
String content = flux.collectList().block().stream().collect(Collectors.joining());
List<ActorFilms> actorFilms = converter.convert(content);
call() return values
After specifying the call() method on ChatClient, there are a few different options for the response type.
-
String content(): returns the String content of the response -
ChatResponse chatResponse(): returns theChatResponseobject that contains multiple generations and also metadata about the response, for example how many token were used to create the response. -
ChatClientResponse chatClientResponse(): returns aChatClientResponseobject that contains theChatResponseobject and the ChatClient execution context, giving you access to additional data used during the execution of advisors (e.g. the relevant documents retrieved in a RAG flow). -
entity()to return a Java type-
entity(ParameterizedTypeReference<T> type): used to return aCollectionof entity types. -
entity(Class<T> type): used to return a specific entity type. -
entity(StructuredOutputConverter<T> structuredOutputConverter): used to specify an instance of aStructuredOutputConverterto convert aStringto an entity type. -
entity(ParameterizedTypeReference<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration. -
entity(Class<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration. -
entity(StructuredOutputConverter<T> converter, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.
-
-
responseEntity()to return both theChatResponseand a Java type. This is useful when you need access to both the complete AI model response (with metadata and generations) and the structured output entity in a single call.-
responseEntity(Class<T> type): used to return aResponseEntitycontaining both the completeChatResponseobject and a specific entity type. -
responseEntity(Class<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration. -
responseEntity(ParameterizedTypeReference<T> type): used to return aResponseEntitycontaining both the completeChatResponseobject and aCollectionof entity types. -
responseEntity(ParameterizedTypeReference<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration. -
responseEntity(StructuredOutputConverter<T> structuredOutputConverter): used to return aResponseEntitycontaining both the completeChatResponseobject and an entity converted using a specifiedStructuredOutputConverter. -
responseEntity(StructuredOutputConverter<T> converter, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.
-
You can also invoke the stream() method instead of call().
Calling the call() method does not actually trigger the AI model execution.
Instead, it only instructs Spring AI whether to use synchronous or streaming calls.
The actual AI model invocation occurs when methods such as content(), chatResponse(), and responseEntity() are called.
|
stream() return values
After specifying the stream() method on ChatClient, there are a few options for the response type:
-
Flux<String> content(): Returns aFluxof the string being generated by the AI model. -
Flux<ChatResponse> chatResponse(): Returns aFluxof theChatResponseobject, which contains additional metadata about the response. -
Flux<ChatClientResponse> chatClientResponse(): returns aFluxof theChatClientResponseobject that contains theChatResponseobject and the ChatClient execution context, giving you access to additional data used during the execution of advisors (e.g. the relevant documents retrieved in a RAG flow).