This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0!

Responses and Streaming

ChatClient Responses

The ChatClient API offers several ways to format the response from the AI Model using the fluent API.

Returning a ChatResponse

The response from the AI model is a rich structure defined by the type ChatResponse. It includes metadata about how the response was generated and can also contain multiple responses, known as Generations, each with its own metadata. The metadata includes the number of tokens (each token is approximately 3/4 of a word) used to create the response. This information is important because hosted AI models charge based on the number of tokens used per request.

An example to return the ChatResponse object that contains the metadata is shown below by invoking chatResponse() after the call() method.

ChatResponse chatResponse = chatClient.prompt()
    .user("Tell me a joke")
    .call()
    .chatResponse();

Finish Reason Categories

Each Generation reports why the model stopped, under the name its provider uses: stop, end_turn, STOP, content_filter, refusal, SAFETY and so on. getFinishReason() returns that raw name. getFinishReasonCategory() maps it onto a FinishReasonCategory that is the same for every provider:

Category Meaning

COMPLETED

The model finished its answer.

TRUNCATED

A length or token limit cut the answer short.

TOOL_CALL

The model asked for a tool to be called.

FILTERED

A safety filter stopped the answer, or the model refused to answer.

OTHER

A reason with no meaning that providers share, or one that Spring AI does not know.

UNSPECIFIED

The provider gave no reason.

The category is read next to the raw finish reason:

ChatResponse chatResponse = chatClient.prompt()
    .user("Tell me a joke")
    .call()
    .chatResponse();

ChatGenerationMetadata metadata = chatResponse.getResult().getMetadata();
if (metadata.getFinishReasonCategory() == FinishReasonCategory.FILTERED) {
    // A safety filter stopped the answer, or the model refused to answer.
    String reason = metadata.getFinishReason();
}

ChatResponse.hasFinishReasonCategory(…​) tells whether any generation of the response finished in a category.

Each chat model classifies its own finish reasons, as it knows what the names of its provider mean and sees the whole provider response. This also covers a stop that the finish reason alone does not tell:

  • OpenAI reports a refusal of the model in a field of its own, with an ordinary stop. The generation is FILTERED, and the refusal is in the refusal metadata of the message.

  • Gemini and Ollama report an ordinary stop when the model asks for a tool. The generation is TOOL_CALL.

A custom ChatModel sets the category with ChatGenerationMetadata.builder().finishReasonCategory(…​). For a generation that carries no category, it is derived from the raw finish reason by its common names, such as stop, length and content_filter.

On a stream, each chunk is classified by what that chunk holds. A stop that the provider spreads over several chunks, such as a refusal of OpenAI or a tool call of Gemini that is followed by a chunk with the finish reason, is therefore not the category of the last chunk, nor of the aggregated response.

Returning an Entity

You often want to return an entity class that is mapped from the returned String. The entity() method provides this functionality.

For example, given the Java record:

record ActorFilms(String actor, List<String> movies) {}

You can easily map the AI model’s output to this record using the entity() method, as shown below:

ActorFilms actorFilms = chatClient.prompt()
    .user("Generate the filmography for a random actor.")
    .call()
    .entity(ActorFilms.class);

There is also an overloaded entity method with the signature entity(ParameterizedTypeReference<T> type) that lets you specify types such as generic Lists:

List<ActorFilms> actorFilms = chatClient.prompt()
    .user("Generate the filmography of 5 movies for Tom Hanks and Bill Murray.")
    .call()
    .entity(new ParameterizedTypeReference<List<ActorFilms>>() {});

Reliability Switches: EntityParamSpec

All entity() overloads accept an optional Consumer<EntityParamSpec> that enables two independent, composable behaviors:

  • validateSchema() — validate the JSON response against the entity schema and automatically retry with error feedback on failure.

  • useProviderStructuredOutput() — send the schema to the provider as an API-level constraint instead of prompt text.

ActorFilms actorFilms = chatClient.prompt()
    .user("Generate the filmography for a random actor.")
    .call()
    .entity(ActorFilms.class, spec -> spec
        .useProviderStructuredOutput()
        .validateSchema());

For full details on these switches, supported providers, limitations, and the lower-level converter API, see the Structured Output reference.

When a Safety Filter Stops the Response

A response that a safety filter stopped, or in which the model refused to answer, is usually empty, cut short, or a refusal in plain words. It is not the entity that was asked for, and parsing it would fail with an error that does not tell why. When the generation that entity() converts finished in the FILTERED category, and its text is empty or cannot be converted, entity() throws a ContentFilteredException:

try {
    ActorFilms actorFilms = chatClient.prompt()
        .user("Generate the filmography for a random actor.")
        .call()
        .entity(ActorFilms.class);
}
catch (ContentFilteredException ex) {
    String reason = ex.getFinishReason(); // as the provider named it, for example "content_filter"
    String text = ex.getText();           // what the provider returned, possibly empty
}

The parsing error, if there was one, is the cause of the exception. A filtered response that converts all the same is returned as an entity.

The refusal of a guardrail advisor is reported the same way, with a GuardrailRefusalException, which is a ContentFilteredException (see Advisors).

responseEntity() does not throw for a filtered response. It returns the response with a null entity, and the finish reason is read from the response. content(), chatResponse() and streaming return the response as the provider left it.

Streaming Responses

The stream() method lets you get an asynchronous response as shown below:

Flux<String> output = chatClient.prompt()
    .user("Tell me a joke")
    .stream()
    .content();

You can also stream the ChatResponse using the method Flux<ChatResponse> chatResponse().

The stream() method has no entity() method. To get an entity from a stream, aggregate the response and convert it with a Structured Output Converter, as shown below. This also demonstrates the use of parameters in the fluent API that will be discussed in more detail in a later section of the documentation.

var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<List<ActorFilms>>() {});

Flux<String> flux = chatClient.prompt()
    .user(u -> u.text("""
                        Generate the filmography for a random actor.
                        {format}
                      """)
            .param("format", converter.getFormat()))
    .stream()
    .content();

String content = flux.collectList().block().stream().collect(Collectors.joining());

List<ActorFilms> actorFilms = converter.convert(content);

call() return values

After specifying the call() method on ChatClient, there are a few different options for the response type.

  • String content(): returns the String content of the response

  • ChatResponse chatResponse(): returns the ChatResponse object that contains multiple generations and also metadata about the response, for example how many token were used to create the response.

  • ChatClientResponse chatClientResponse(): returns a ChatClientResponse object that contains the ChatResponse object and the ChatClient execution context, giving you access to additional data used during the execution of advisors (e.g. the relevant documents retrieved in a RAG flow).

  • entity() to return a Java type

    • entity(ParameterizedTypeReference<T> type): used to return a Collection of entity types.

    • entity(Class<T> type): used to return a specific entity type.

    • entity(StructuredOutputConverter<T> structuredOutputConverter): used to specify an instance of a StructuredOutputConverter to convert a String to an entity type.

    • entity(ParameterizedTypeReference<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

    • entity(Class<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

    • entity(StructuredOutputConverter<T> converter, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

  • responseEntity() to return both the ChatResponse and a Java type. This is useful when you need access to both the complete AI model response (with metadata and generations) and the structured output entity in a single call.

    • responseEntity(Class<T> type): used to return a ResponseEntity containing both the complete ChatResponse object and a specific entity type.

    • responseEntity(Class<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

    • responseEntity(ParameterizedTypeReference<T> type): used to return a ResponseEntity containing both the complete ChatResponse object and a Collection of entity types.

    • responseEntity(ParameterizedTypeReference<T> type, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

    • responseEntity(StructuredOutputConverter<T> structuredOutputConverter): used to return a ResponseEntity containing both the complete ChatResponse object and an entity converted using a specified StructuredOutputConverter.

    • responseEntity(StructuredOutputConverter<T> converter, Consumer<EntityParamSpec> spec): like above, with optional EntityParamSpec configuration.

You can also invoke the stream() method instead of call().

Calling the call() method does not actually trigger the AI model execution. Instead, it only instructs Spring AI whether to use synchronous or streaming calls. The actual AI model invocation occurs when methods such as content(), chatResponse(), and responseEntity() are called.

stream() return values

After specifying the stream() method on ChatClient, there are a few options for the response type:

  • Flux<String> content(): Returns a Flux of the string being generated by the AI model.

  • Flux<ChatResponse> chatResponse(): Returns a Flux of the ChatResponse object, which contains additional metadata about the response.

  • Flux<ChatClientResponse> chatClientResponse(): returns a Flux of the ChatClientResponse object that contains the ChatResponse object and the ChatClient execution context, giving you access to additional data used during the execution of advisors (e.g. the relevant documents retrieved in a RAG flow).