This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0!

Multimodality API

"All things that are naturally connected ought to be taught in combination" - John Amos Comenius, "Orbis Sensualium Pictus", 1658

Humans process knowledge, simultaneously across multiple modes of data inputs. The way we learn, our experiences are all multimodal. We don’t have just vision, just audio and just text.

Contrary to those principles, the Machine Learning was often focused on specialized models tailored to process a single modality. For instance, we developed audio models for tasks like text-to-speech or speech-to-text, and computer vision models for tasks such as object detection and classification.

However, a new wave of multimodal large language models starts to emerge. Examples include OpenAI’s GPT, Google’s Gemini, Anthropic’s Claude, and open source offerings Llama, LLaVA and BakLLaVA are able to accept multiple inputs, including text images, audio and video and generate text responses by integrating these inputs.

The multimodal large language model (LLM) features enable the models to process and generate text in conjunction with other modalities such as images, audio, or video.

Spring AI Multimodality

Multimodality refers to a model’s ability to simultaneously understand and process information from various sources, including text, images, audio, and other data formats.

The Spring AI Message API provides all necessary abstractions to support multimodal LLMs.

The Message API Message extends Content, which holds the text and metadata, and adds the MessageType: SYSTEM, USER, ASSISTANT or TOOL. AbstractMessage holds the ordered MessageParts: TextPart, ReasoningPart, ToolCallPart, ToolResultPart, MediaPart and UnknownPart. SystemMessage, ToolResponseMessage, UserMessage and AssistantMessage extend AbstractMessage, and UserMessage and AssistantMessage also implement MediaContent. «enum»MessageTypeSYSTEM, USER,ASSISTANT, TOOL «sealed interface»MessagePartTextPartReasoningPartToolCallPartToolResultPartMediaPartUnknownPart ContentgetText(): StringgetMetadata(): Map<String, Object> MessagegetMessageType(): MessageType MediaContentgetMedia(): List<Media> «abstract»AbstractMessagegetParts(): List<MessagePart> type parts 0..* SystemMessage ToolResponseMessagegetResponses() UserMessagegetMedia() AssistantMessagegetToolCalls()getReasoning()getMedia()

A UserMessage holds its content as an ordered list of parts: TextPart entries for the text, and MediaPart entries for images, audio, video and documents. getText() and getMedia() are views over these parts (see Message Parts), and MediaContent is the interface that declares getMedia(). The MimeType specifies the modality type. Depending on the used LLMs, the Media data field can be either the raw media content as a Resource object or a URI to the content.

System messages carry no media. An AssistantMessage holds its content as ordered parts as well (text, reasoning, tool calls and media), and getMedia() returns its media parts. A chat model that returns media, such as an image generated by Gemini, puts it there. For other non-text output, use one of the dedicated single-modality models, such as the image or text-to-speech models.

For example, we can take the following picture (multimodal.test.png) as an input and ask the LLM to explain what it sees.

Multimodal Test Image

For most of the multimodal LLMs, the Spring AI code would look something like this:

  • MediaPart

  • media(…​)

var imageResource = new ClassPathResource("/multimodal.test.png");

var userMessage = UserMessage.builder()
    .part(TextPart.of("Explain what do you see in this picture?"))
    .part(MediaPart.of(new Media(MimeTypeUtils.IMAGE_PNG, imageResource)))
    .build();

ChatResponse response = chatModel.call(new Prompt(userMessage));
var imageResource = new ClassPathResource("/multimodal.test.png");

var userMessage = UserMessage.builder()
    .text("Explain what do you see in this picture?")
    .media(new Media(MimeTypeUtils.IMAGE_PNG, imageResource))
    .build();

ChatResponse response = chatModel.call(new Prompt(userMessage));

or with the fluent ChatClient API:

  • MediaPart

  • media(…​)

var userMessage = UserMessage.builder()
    .part(TextPart.of("Explain what do you see on this picture?"))
    .part(MediaPart.of(new Media(MimeTypeUtils.IMAGE_PNG, new ClassPathResource("/multimodal.test.png"))))
    .build();

String response = ChatClient.create(chatModel)
    .prompt()
    .messages(userMessage)
    .call()
    .content();
String response = ChatClient.create(chatModel).prompt()
    .user(u -> u.text("Explain what do you see on this picture?")
        .media(MimeTypeUtils.IMAGE_PNG, new ClassPathResource("/multimodal.test.png")))
    .call()
    .content();

The MediaPart version adds the parts in the order they should reach the model, so text and media can interleave. ChatClient has no part(…​) method on its user spec, so pass the message with messages(…​).

Each version produces a response like:

This is an image of a fruit bowl with a simple design. The bowl is made of metal with curved wire edges that create an open structure, allowing the fruit to be visible from all angles. Inside the bowl, there are two yellow bananas resting on top of what appears to be a red apple. The bananas are slightly overripe, as indicated by the brown spots on their peels. The bowl has a metal ring at the top, likely to serve as a handle for carrying. The bowl is placed on a flat surface with a neutral-colored background that provides a clear view of the fruit inside.

Spring AI provides multimodal support for the following chat models: