The following properties are available for Kafka consumers only and must be prefixed with spring.cloud.stream.kafka.bindings.<channelName>.consumer..

When an application starts, the initial position in each assigned partition depends on two properties startOffset and resetOffsets. If resetOffsets is false, normal Kafka consumer auto.offset.reset semantics apply. i.e. If there is no committed offset for a partition for the binding’s consumer group, the position is earliest or latest. By default, bindings with an explicit group use earliest, and anonymous bindings (with no group) use latest. These defaults can be overridden by setting the startOffset binding property. There will be no committed offset(s) the first time the binding is started with a particular group. The other condition where no committed offset exists is if the offset has been expired. With modern brokers (since 2.1), and default broker properties, the offsets are expired 7 days after the last member leaves the group. See the offsets.retention.minutes broker property for more information.

When resetOffsets is true, the binder applies similar semantics to those that apply when there is no committed offset on the broker, as if this binding has never consumed from the topic; i.e. any current committed offset is ignored.

Following are two use cases when this might be used.

For more control over topic offsets, see Using a KafkaBindingRebalanceListener; when a listener is provided, resetOffsets should not be set to true, otherwise, that will cause an error.

Starting with version 3.0, when spring.cloud.stream.binding.<name>.consumer.batch-mode is set to true, all of the records received by polling the Kafka Consumer will be presented as a List<?> to the listener method. Otherwise, the method will be called with one record at a time. The size of the batch is controlled by Kafka consumer properties max.poll.records, fetch.min.bytes, fetch.max.wait.ms; refer to the Kafka documentation for more information.

Starting with version 4.0.2, the binder supports DLQ capabilities when consuming in batch mode. Keep in mind that, when using DLQ on a consumer binding that is in batch mode, all the records received from the previous poll will be delivered to the DLQ topic.

The following properties are available for Kafka producers only and must be prefixed with spring.cloud.stream.kafka.bindings.<channelName>.producer..

In this section, we show the use of the preceding properties for specific scenarios.

By default, records are published to the Dead-Letter topic using the same partition as the original record. This means the Dead-Letter topic must have at least as many partitions as the original record.

To change this behavior, add a DlqPartitionFunction implementation as a @Bean to the application context. Only one such bean can be present. The function is provided with the consumer group, the failed ConsumerRecord and the exception. For example, if you always want to route to partition 0, you might use:

It is also possible to define a custom name for the DLQ topic. In order to do so, create an implementation of DlqDestinationResolver as a @Bean to the application context. When the binder detects such a bean, that takes precedence, otherwise it will use the dlqName property. If neither of these are found, it will default to error.<destination>.<group>. Here is an example of DlqDestinationResolver as a @Bean.

One important thing to keep in mind when providing an implementation for DlqDestinationResolver is that the provisioner in the binder will not auto create topics for the application. This is because there is no way for the binder to infer the names of all the DLQ topics the implementation might send to. Therefore, if you provide DLQ names using this strategy, it is the application’s responsibility to ensure that those topics are created beforehand.

Because the framework cannot anticipate how users would want to dispose of dead-lettered messages, it does not provide any standard mechanism to handle them. If the reason for the dead-lettering is transient, you may wish to route the messages back to the original topic. However, if the problem is a permanent issue, that could cause an infinite loop. The sample Spring Boot application within this topic is an example of how to route those messages back to the original topic, but it moves them to a “parking lot” topic after three attempts. The application is another spring-cloud-stream application that reads from the dead-letter topic. It exits when no messages are received for 5 seconds.

The examples assume the original destination is so8400out and the consumer group is so8400.

There are a couple of strategies to consider:

The following code listings show the sample application:

Starting with Spring Cloud Stream 3.0.0, Kafka Streams binder allows the applications to be designed and developed using the functional programming style that is available in Java 8. This means that the applications can be concisely represented as a lambda expression of types java.util.function.Function or java.util.function.Consumer.

Let’s take a very basic example.

Albeit simple, this is a complete standalone Spring Boot application that is leveraging Kafka Streams for stream processing. This is a consumer application with no outbound binding and only a single inbound binding. The application consumes data and it simply logs the information from the KStream key and value on the standard output. The application contains the SpringBootApplication annotation and a method that is marked as Bean. The bean method is of type java.util.function.Consumer which is parameterized with KStream. Then in the implementation, we are returning a Consumer object that is essentially a lambda expression. Inside the lambda expression, the code for processing the data is provided.

In this application, there is a single input binding that is of type KStream. The binder creates this binding for the application with a name process-in-0, i.e. the name of the function bean name followed by a dash character (-) and the literal in followed by another dash and then the ordinal position of the parameter. You use this binding name to set other properties such as destination. For example, spring.cloud.stream.bindings.process-in-0.destination=my-topic.

Once built as a uber-jar (e.g., kstream-consumer-app.jar), you can run the above example like the following.

If the applications choose to define the functional beans using Spring’s Component annotation, the binder also supports that model. The above functional bean could be rewritten as below.

Here is another example, where it is a full processor with both input and output bindings. This is the classic word-count example in which the application receives data from a topic, the number of occurrences for each word is then computed in a tumbling time-window.

Here again, this is a complete Spring Boot application. The difference here from the first application is that the bean method is of type java.util.function.Function. The first parameterized type for the Function is for the input KStream and the second one is for the output. In the method body, a lambda expression is provided that is of type Function and as implementation, the actual business logic is given. Similar to the previously discussed Consumer based application, the input binding here is named as process-in-0 by default. For the output, the binding name is automatically also set to process-out-0.

Once built as an uber-jar (e.g., wordcount-processor.jar), you can run the above example like the following.

This application will consume messages from the Kafka topic words and the computed results are published to an output topic counts.

Spring Cloud Stream will ensure that the messages from both the incoming and outgoing topics are automatically bound as KStream objects. As a developer, you can exclusively focus on the business aspects of the code, i.e. writing the logic required in the processor. Setting up Kafka Streams specific configuration required by the Kafka Streams infrastructure is automatically handled by the framework.

The two examples we saw above have a single KStream input binding. In both cases, the bindings received the records from a single topic. If you want to multiplex multiple topics into a single KStream binding, you can provide comma separated Kafka topics as destinations below.

spring.cloud.stream.bindings.process-in-0.destination=topic-1,topic-2,topic-3

In addition, you can also provide topic patterns as destinations if you want to match topics against a regular exression.

spring.cloud.stream.bindings.process-in-0.destination=input.*

Binder allows to have multiple Kafka Streams processors within a single Spring Cloud Stream application. You can have an application as below.

In this case, the binder will create 3 separate Kafka Streams objects with different application ID’s (more on this below). However, if you have more than one processor in the application, you have to tell Spring Cloud Stream, which functions need to be activated. Here is how you activate the functions.

spring.cloud.function.definition: process;anotherProcess;yetAnotherProcess

If you want certain functions to be not activated right away, you can remove that from this list.

This is also true when you have a single Kafka Streams processor and other types of Function beans in the same application that is handled through a different binder (for e.g., a function bean that is based on the regular Kafka Message Channel binder)

Application id is a mandatory property that you need to provide for a Kafka Streams application. Spring Cloud Stream Kafka Streams binder allows you to configure this application id in multiple ways.

If you only have one single processor in the application, then you can set this at the binder level using the following property:

spring.cloud.stream.kafka.streams.binder.applicationId.

As a convenience, if you only have a single processor, you can also use spring.application.name as the property to delegate the application id.

If you have multiple Kafka Streams processors in the application, then you need to set the application id per processor. In the case of the functional model, you can attach it to each function as a property.

For e.g. imagine that you have the following functions.

and

Then you can set the application id for each, using the following binder level properties.

spring.cloud.stream.kafka.streams.binder.functions.process.applicationId

and

spring.cloud.stream.kafka.streams.binder.functions.anotherProcess.applicationId

For function based model also, this approach of setting application id at the binding level will work. However, setting per function at the binder level as we have seen above is much easier if you are using the functional model.

For production deployments, it is highly recommended to explicitly specify the application ID through configuration. This is especially going to be very critical if you are auto scaling your application in which case you need to make sure that you are deploying each instance with the same application ID.

If the application does not provide an application ID, then in that case the binder will auto generate a static application ID for you. This is convenient in development scenarios as it avoids the need for explicitly providing the application ID. The generated application ID in this manner will be static over application restarts. In the case of functional model, the generated application ID will be the function bean name followed by the literal applicationID, for e.g process-applicationID if process if the function bean name.

By default, the binder uses the strategy discussed above to generate the binding name when using the functional style, i.e. <function-bean-name>-<in>|<out>-[0..n], for e.g. process-in-0, process-out-0 etc. If you want to override those binding names, you can do that by specifying the following properties.

spring.cloud.stream.function.bindings.<default binding name>. Default binding name is the original binding name generated by the binder.

For e.g. lets say, you have this function.

Binder will generate bindings with names, process-in-0, process-in-1 and process-out-0. Now, if you want to change them to something else completely, maybe more domain specific binding names, then you can do so as below.

spring.cloud.stream.function.bindings.process-in-0=users

spring.cloud.stream.function.bindings.process-in-0=regions

and

spring.cloud.stream.function.bindings.process-out-0=clicks

After that, you must set all the binding level properties on these new binding names.

Please keep in mind that with the functional programming model described above, adhering to the default binding names make sense in most situations. The only reason you may still want to do this overriding is when you have larger number of configuration properties and you want to map the bindings to something more domain friendly.

When running Kafka Streams applications, you must provide the Kafka broker server information. If you don’t provide this information, the binder expects that you are running the broker at the default localhost:9092. If that is not the case, then you need to override that. There are a couple of ways to do that.

When it comes to the binder level property, it doesn’t matter if you use the broker property provided through the regular Kafka binder - spring.cloud.stream.kafka.binder.brokers. Kafka Streams binder will first check if Kafka Streams binder specific broker property is set (spring.cloud.stream.kafka.streams.binder.brokers) and if not found, it looks for spring.cloud.stream.kafka.binder.brokers.

Keys are always deserialized using native Serdes.

For values, by default, deserialization on the inbound is natively performed by Kafka. Please note that this is a major change on default behavior from previous versions of Kafka Streams binder where the deserialization was done by the framework.

Kafka Streams binder will try to infer matching Serde types by looking at the type signature of java.util.function.Function|Consumer. Here is the order that it matches the Serdes.

If none of the above strategies worked, then the applications must provide the Serde s through configuration. This can be configured in two ways - binding or default.

First the binder will look if a Serde is provided at the binding level. For e.g. if you have the following processor,

then, you can provide a binding level Serde using the following:

If you want the default key/value Serdes to be used for inbound deserialization, you can do so at the binder level.

If you don’t want the native decoding provided by Kafka, you can rely on the message conversion features that Spring Cloud Stream provides. Since native decoding is the default, in order to let Spring Cloud Stream deserialize the inbound value object, you need to explicitly disable native decoding.

For e.g. if you have the same BiFunction processor as above, then spring.cloud.stream.bindings.process-in-0.consumer.nativeDecoding: false You need to disable native decoding for all the inputs individually. Otherwise, native decoding will still be applied for those you do not disable.

By default, Spring Cloud Stream will use application/json as the content type and use an appropriate json message converter. You can use custom message converters by using the following property and an appropriate MessageConverter bean.

Outbound serialization pretty much follows the same rules as above for inbound deserialization. As with the inbound deserialization, one major change from the previous versions of Spring Cloud Stream is that the serialization on the outbound is handled by Kafka natively. Before 3.0 versions of the binder, this was done by the framework itself.

Keys on the outbound are always serialized by Kafka using a matching Serde that is inferred by the binder. If it can’t infer the type of the key, then that needs to be specified using configuration.

Value serdes are inferred using the same rules used for inbound deserialization. First it matches to see if the outbound type is from a provided bean in the application. If not, it checks to see if it matches with a Serde exposed by Kafka such as - Integer, Long, Short, Double, Float, byte[], UUID and String. If that doesnt’t work, then it falls back to JsonSerde provided by the Spring Kafka project, but first look at the default Serde configuration to see if there is a match. Keep in mind that all these happen transparently to the application. If none of these work, then the user has to provide the Serde to use by configuration.

Lets say you are using the same BiFunction processor as above. Then you can configure outbound key/value Serdes as following.

If Serde inference fails, and no binding level Serdes are provided, then the binder falls back to the JsonSerde, but look at the default Serdes for a match.

Default serdes are configured in the same way as above where it is described under deserialization.

spring.cloud.stream.kafka.streams.binder.configuration.default.key.serde spring.cloud.stream.kafka.streams.binder.configuration.default.value.serde

If your application uses the branching feature and has multiple output bindings, then these have to be configured per binding. Once again, if the binder is capable of inferring the Serde types, you don’t need to do this configuration.

If you don’t want the native encoding provided by Kafka, but want to use the framework provided message conversion, then you need to explicitly disable native encoding since since native encoding is the default. For e.g. if you have the same BiFunction processor as above, then spring.cloud.stream.bindings.process-out-0.producer.nativeEncoding: false You need to disable native encoding for all the output individually in the case of branching. Otherwise, native encoding will still be applied for those you don’t disable.

When conversion is done by Spring Cloud Stream, by default, it will use application/json as the content type and use an appropriate json message converter. You can use custom message converters by using the following property and a corresponding MessageConverter bean.

When native encoding/decoding is disabled, binder will not do any inference as in the case of native Serdes. Applications need to explicitly provide all the configuration options. For that reason, it is generally advised to stay with the default options for de/serialization and stick with native de/serialization provided by Kafka Streams when you write Spring Cloud Stream Kafka Streams applications. The one scenario in which you must use message conversion capabilities provided by the framework is when your upstream producer is using a specific serialization strategy. In that case, you want to use a matching deserialization strategy as native mechanisms may fail. When relying on the default Serde mechanism, the applications must ensure that the binder has a way forward with correctly map the inbound and outbound with a proper Serde, as otherwise things might fail.

It is worth to mention that the data de/serialization approaches outlined above are only applicable on the edges of your processors, i.e. - inbound and outbound. Your business logic might still need to call Kafka Streams API’s that explicitly need Serde objects. Those are still the responsibility of the application and must be handled accordingly by the developer.

Kafka Streams binder allows to specify the deserialization exception handlers above using the following property.

or

In addition to the above two deserialization exception handlers, the binder also provides a third one for sending the erroneous records (poison pills) to a DLQ (dead letter queue) topic. Here is how you enable this DLQ exception handler.

When the above property is set, all the records in deserialization error are automatically sent to the DLQ topic.

You can set the topic name where the DLQ messages are published as below.

You can provide an implementation for DlqDestinationResolver which is a functional interface. DlqDestinationResolver takes ConsumerRecord and the exception as inputs and then allows to specify a topic name as the output. By gaining access to the Kafka ConsumerRecord, the header records can be introspected in the implementation of the BiFunction.

Here is an example of providing an implementation for DlqDestinationResolver.

One important thing to keep in mind when providing an implementation for DlqDestinationResolver is that the provisioner in the binder will not auto create topics for the application. This is because there is no way for the binder to infer the names of all the DLQ topics the implementation might send to. Therefore, if you provide DLQ names using this strategy, it is the application’s responsibility to ensure that those topics are created beforehand.

If DlqDestinationResolver is present in the application as a bean, that takes higher precedence. If you do not want to follow this approach and rather provide a static DLQ name using configuration, you can set the following property.

If this is set, then the error records are sent to the topic custom-dlq. If the application is not using either of the above strategies, then it will create a DLQ topic with the name error.<input-topic-name>.<application-id>. For instance, if your binding’s destination topic is inputTopic and the application ID is process-applicationId, then the default DLQ topic is error.inputTopic.process-applicationId. It is always recommended to explicitly create a DLQ topic for each input binding if it is your intention to enable DLQ.

The property spring.cloud.stream.kafka.streams.binder.deserializationExceptionHandler is applicable for the entire application. This implies that if there are multiple functions in the same application, this property is applied to all of them. However, if you have multiple processors or multiple input bindings within a single processor, then you can use the finer-grained DLQ control that the binder provides per input consumer binding.

If you have the following processor,

and you only want to enable DLQ on the first input binding and skipAndContinue on the second binding, then you can do so on the consumer as below.

spring.cloud.stream.kafka.streams.bindings.process-in-0.consumer.deserializationExceptionHandler: sendToDlq spring.cloud.stream.kafka.streams.bindings.process-in-1.consumer.deserializationExceptionHandler: skipAndContinue

Setting deserialization exception handlers this way has a higher precedence than setting at the binder level.

By default, records are published to the Dead-Letter topic using the same partition as the original record. This means the Dead-Letter topic must have at least as many partitions as the original record.

To change this behavior, add a DlqPartitionFunction implementation as a @Bean to the application context. Only one such bean can be present. The function is provided with the consumer group (which is the same as the application ID in most situations), the failed ConsumerRecord and the exception. For example, if you always want to route to partition 0, you might use:

A couple of things to keep in mind when using the exception handling feature in Kafka Streams binder.

Unlike the support for deserialization exception handlers as described above, the binder does not provide such first class mechanisms for handling production exceptions. However, you still can configure production exception handlers using the StreamsBuilderFactoryBean customizer which you can find more details about, in a subsequent section below.

Use the following API method to retrieve the KeyQueryMetadata object associated with the combination of given store and key.

Use the following API method to retrieve the KakfaStreams object associated with the combination of given store and key.

Sometimes it is necessary that you need to fine tune the store query parameters before querying the store through InteractiveQueryService. For this purpose, starting with the 4.0.1 version of the binder, you can provide a bean for StoreQueryParametersCustomizer which is a functional interface with a customize method that takes a StoreQueryParameter as the argument. Here is its method signature.

Using this approach, applications can further customize the StoreQueryParameters such as enabling stale stores.

When this bean is present in this application, InteractiveQueryService will call its customize method before querying the state store.

As mentioned above, the binder does not provide a first class way to register global state stores as a feature. For that, you need to use the customizer. Here is how that can be done.

Again, if you have multiple processors, you want to attach the global state store to the right StreamsBuilder by filtering out the other StreamsBuilderFactoryBean objects using the application id as outlined above.

In the error handling section, we indicated that the binder does not provide a first class way to deal with production exceptions. Though that is the case, you can still use the StreamsBuilderFacotryBean customizer to register production exception handlers. See below.

Once again, if you have multiple processors, you may want to set it appropriately against the correct StreamsBuilderFactoryBean. You may also add such production exception handlers using the configuration property (See below for more on that), but this is an option if you choose to go with a programmatic approach.

The following properties are available at the binder level and must be prefixed with spring.cloud.stream.kafka.streams.binder. Any Kafka binder provided properties re-used in Kafka Streams binder must be prefixed with spring.cloud.stream.kafka.streams.binder instead of spring.cloud.stream.kafka.binder. The only exception to this rule is when defining the Kafka bootstrap server property in which case either prefix works.

For more information about all the properties that may go into streams configuration, see StreamsConfig JavaDocs in Apache Kafka Streams docs. All configuration that you can set from StreamsConfig can be set through this. When using this property, it is applicable against the entire application since this is a binder level property. If you have more than processors in the application, all of them will acquire these properties. In the case of properties like application.id, this will become problematic and therefore you have to carefully examine how the properties from StreamsConfig are mapped using this binder level configuration property.

The following properties are only available for Kafka Streams producers and must be prefixed with spring.cloud.stream.kafka.streams.bindings.<binding name>.producer. For convenience, if there are multiple output bindings and they all require a common value, that can be configured by using the prefix spring.cloud.stream.kafka.streams.default.producer..

The following properties are available for Kafka Streams consumers and must be prefixed with spring.cloud.stream.kafka.streams.bindings.<binding-name>.consumer. For convenience, if there are multiple input bindings and they all require a common value, that can be configured by using the prefix spring.cloud.stream.kafka.streams.default.consumer..

Note: Using resetOffsets on the consumer does not have any effect on Kafka Streams binder. Unlike the message channel based binder, Kafka Streams binder does not seek to beginning or end on demand.

In Kafka Streams, you can control of the number of threads a processor can create using the num.stream.threads property. This, you can do using the various configuration options described above under binder, functions, producer or consumer level. You can also use the concurrency property that core Spring Cloud Stream provides for this purpose. When using this, you need to use it on the consumer. When you have more than one input binding, set this on the first input binding. For e.g. when setting spring.cloud.stream.bindings.process-in-0.consumer.concurrency, it will be translated as num.stream.threads by the binder. If you have multiple processors and one processor defines binding level concurrency, but not the others, those ones with no binding level concurrency will default back to the binder wide property specified through spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads. If this binder configuration is not available, then the application will use the default set by Kafka Streams.

As a developer, I want to write a consumer application that processes records from a Kafka topic. However, if some error occurs in processing, I don’t want the application to stop completely. Instead, I want to send the record in error to a DLT (Dead-Letter-Topic) and then continue processing new records.

The solution for this problem is to use the DLQ feature in Spring Cloud Stream. For the purposes of this discussion, let us assume that the following is our processor function.

This is a very trivial function that throws an exception for all the records that it processes, but you can take this function and extend it to any other similar situations.

In order to send the records in error to a DLT, we need to provide the following configuration.

In order to activate DLQ, the application must provide a group name. Anonymous consumers cannot use the DLQ facilities. We also need to enable DLQ by setting the enableDLQ property on the Kafka consumer binding to true. Finally, we can optionally provide the DLT name by providing the dlqName on Kafka consumer binding, which otherwise default to error.input-topic.my-group in this case.

Note that in the example consumer provided above, the type of the payload is byte[]. By default, the DLQ producer in Kafka binder expects the payload of type byte[]. If that is not the case, then we need to provide the configuration for proper serializer. For example, let us re-write the consumer function as below:

Now, we need to tell Spring Cloud Stream, how we want to serialize the data when writing to the DLT. Here is the modified configuration for this scenario:

This is similar to the recipe above, but as a developer I would like to configure the way retries are handled.

If you followed the above recipe, then you get the default retry options built into the Kafka binder when the processing encounters an error.

By default, the binder retires for a maximum of 3 attempts with a one second initial delay, 2.0 multiplier with each back off with a max delay of 10 seconds. You can change all these configurations as below:

If you want, you can also provide a list of retryable exceptions by providing a map of boolean values. For example,

By default, any exceptions not listed in the map above will be retried. If that is not desired, then you can disable that by providing,

You can also provide your own RetryTemplate and mark it as @StreamRetryTemplate which will be scanned and used by the binder. This is useful when you want more sophisticated retry strategies and policies.

If you have multiple @StreamRetryTemplate beans, then you can specify which one your binding wants by using the property,

I have a processor that encounters a deserialization exception in Kafka consumer. I would expect that the Spring Cloud Stream DLQ mechanism will catch that scenario, but it does not. How can I handle this?

The normal DLQ mechanism offered by Spring Cloud Stream will not help when Kafka consumer throws an irrecoverable deserialization exception. This is because, this exception happens even before the consumer’s poll() method returns. Spring for Apache Kafka project offers some great ways to help the binder with this situation. Let us explore those.

Assuming this is our function:

It is a trivial function that takes a String parameter.

We want to bypass the message converters provided by Spring Cloud Stream and want to use native deserializers instead. In the case of String types, it does not make much sense, but for more complex types like AVRO etc. you have to rely on external deserializers and therefore want to delegate the conversion to Kafka.

Now when the consumer receives the data, let us assume that there is a bad record that causes a deserialization error, maybe someone passed an Integer instead of a String for example. In that case, if you don’t do something in the application, the exception will be propagated through the chain and your application will exit eventually.

In order to handle this, you can add a ListenerContainerCustomizer @Bean that configures a DefaultErrorHandler. This DefaultErrorHandler is configured with a DeadLetterPublishingRecoverer. We also need to configure an ErrorHandlingDeserializer for the consumer. That sounds like a lot of complex things, but in reality, it boils down to these 3 beans in this case.

Let us analyze each of them. The first one is the ListenerContainerCustomizer bean that takes a DefaultErrorHandler. The container is now customized with that particular error handler. You can learn more about container customization here.

The second bean is the DefaultErrorHandler that is configured with a publishing to a DLT. See here for more details on DefaultErrorHandler.

The third bean is the DeadLetterPublishingRecoverer that is ultimately responsible for sending to the DLT. By default, the DLT topic is named as the ORIGINAL_TOPIC_NAME.DLT. You can change that though. See the docs for more details.

We also need to configure an ErrorHandlingDeserializer through application config.

The ErrorHandlingDeserializer delegates to the actual deserializer. In case of errors, it sets key/value of the record to be null and includes the raw bytes of the message. It then sets the exception in a header and passes this record to the listener, which then calls the registered error handler.

Following is the configuration required:

We are providing the ErrorHandlingDeserializer through the configuration property on the binding. We are also indicating that the actual deserializer to delegate is the StringDeserializer.

Keep in mind that none of the dlq properties above are relevant for the discussions in this recipe. They are purely meant for addressing any application level errors only.

I want to write a Spring Cloud Stream Kafka consumer application and not sure about how it manages Kafka consumer offsets. Can you explain?

We encourage you read the docs section on this to get a thorough understanding on it.

Here is it in a gist:

Kafka supports two types of offsets to start with by default - earliest and latest. Their semantics are self-explanatory from their names.

Assuming you are running the consumer for the first time. If you miss the group.id in your Spring Cloud Stream application, then it becomes an anonymous consumer. Whenever, you have an anonymous consumer, in that case, Spring Cloud Stream application by default will start from the latest available offset in the topic partition. On the other hand, if you explicitly specify a group.id, then by default, the Spring Cloud Stream application will start from the earliest available offset in the topic partition.

In both cases above (consumers with explicit groups and anonymous groups), the starting offset can be switched around by using the property spring.cloud.stream.kafka.bindings.<binding-name>.consumer.startOffset and setting it to either earliest or latest.

Now, assume that you already ran the consumer before and now starting it again. In this case, the starting offset semantics in the above case do not apply as the consumer finds an already committed offset for the consumer group (In the case of an anonymous consumer, although the application does not provide a group.id, the binder will auto generate one for you). It simply picks up from the last committed offset onward. This is true, even when you have a startOffset value provided.

However, you can override the default behavior where the consumer starts from the last committed offset by using the resetOffsets property. In order to do that, set the property spring.cloud.stream.kafka.bindings.<binding-name>.consumer.resetOffsets to true (which is false by default). Then make sure you provide the startOffset value (either earliest or latest). When you do that and then start the consumer application, each time you start, it starts as if this is starting for the first time and ignore any committed offsets for the partition.

Using Kafka binder, I know that it can set the offset to either earliest or latest, but I have a requirement to seek the offset to something in the middle, an arbitrary offset. Is there a way to achieve this using Spring Cloud Stream Kafka binder?

Previously we saw how Kafka binder allows you to tackle basic offset management. By default, the binder does not allow you to rewind to an arbitrary offset, at least through the mechanism we saw in that recipe. However, there are some low-level strategies that the binder provides to achieve this use case. Let’s explore them.

First of all, when you want to reset to an arbitrary offset other than earliest or latest, make sure to leave the resetOffsets configuration to its defaults, which is false. Then you have to provide a custom bean of type KafkaBindingRebalanceListener, which will be injected into all consumer bindings. It is an interface that comes with a few default methods, but here is the method that we are interested in:

Let us look at the details.

In essence, this method will be invoked each time during the initial assignment for a topic partition or after a rebalance. For better illustration, let us assume that our topic is foo and it has 4 partitions. Initially, we are only starting a single consumer in the group and this consumer will consume from all partitions. When the consumer starts for the first time, all 4 partitions are getting initially assigned. However, we do not want to start the partitions to consume at the defaults (earliest since we define a group), rather for each partition, we want them to consume after seeking to arbitrary offsets. Imagine that you have a business case to consume from certain offsets as below.

This could be achieved by implementing the above method as below.

This is just a rudimentary implementation. Real world use cases are much more complex than this and you need to adjust accordingly, but this certainly gives you a basic sketch. When consumer seek fails, it may throw some runtime exceptions and you need to decide what to do in those cases.

When we add a second consumer, a rebalance will occur and some partitions will be moved around. Let’s say that the new consumer gets partitions 2 and 3. When this new Spring Cloud Stream consumer calls this onPartitionsAssigned method, it will see that this is the initial assignment for partition 2 and 3 on this consumer. Therefore, it will do the seek operation because of the conditional check on the initial argument. In the case of the first consumer, it now only has partitions 0 and 1 However, for this consumer it was simply a rebalance event and not considered as an intial assignment. Thus, it will not re-seek to the given offsets because of the conditional check on the initial argument.

Using Kafka binder, I want to manually acknowledge messages in my consumer. How do I do that?

By default, Kafka binder delegates to the default commit settings in Spring for Apache Kafka project. The default ackMode in Spring Kafka is batch. See here for more details on that.

There are situations in which you want to disable this default commit behavior and rely on manual commits. Following steps allow you to do that.

Set the property spring.cloud.stream.kafka.bindings.<binding-name>.consumer.ackMode to either MANUAL or MANUAL_IMMEDIATE. When it is set like that, then there will be a header called kafka_acknowledgment (from KafkaHeaders.ACKNOWLEDGMENT) present in the message received by the consumer method.

For example, imagine this as your consumer method.

Then you set the property spring.cloud.stream.kafka.bindings.myConsumer-in-0.consumer.ackMode to MANUAL or MANUAL_IMMEDIATE.

Spring Cloud Stream creates default bindings based on the function definition and signature, but how do I override these to more domain friendly names?

Assume that following is your function signature.

By default, Spring Cloud Stream will create the bindings as below.

You can override these bindings to something by using the following properties.

After this, all binding properties must be made on the new names, my-transformer-in and my-transformer-out.

Here is another example with Kafka Streams and multiple inputs.

By default, Spring Cloud Stream will create three different binding names for this function.

You have to use these binding names each time you want to set some configuration on these bindings. You don’t like that, and you want to use more domain-friendly and readable binding names, for example, something like.

You can easily do that by simply setting these three properties

Once you do that, it overrides the default binding names and any properties that you want to set on them must be on these new binding names.

I need to send a key along with the payload of the record, is there a way to do that in Spring Cloud Stream?

It is often necessary that you want to send associative data structure like a map as the record with a key and value. Spring Cloud Stream allows you to do that in a straightforward manner. Following is a basic blueprint for doing this, but you may want to adapt it to your paricular use case.

Here is sample producer method (aka Supplier).

This is a trivial function that sends a message with a String payload, but also with a key. Note that we set the key as a message header using KafkaHeaders.MESSAGE_KEY.

If you want to change the key from the default kafka_messageKey, then in the configuration, we need to specify this property:

Please note that we use the binding name supplier-out-0 since that is our function name, please update accordingly.

Then, we use this new key when we produce the message.

Instead of using the message converters in Spring Cloud Stream, I want to use native Serializer and Deserializer in Kafka. By default, Spring Cloud Stream takes care of this conversion using its internal built-in message converters. How can I bypass this and delegate the responsibility to Kafka?

This is really easy to do.

All you have to do is to provide the following property to enable native serialization.

Then, you need to also set the serializers. There are a couple of ways to do this.

or using the binder configuration.

When using the binder way, it is applied against all the bindings whereas setting them at the bindings are per binding.

On the deserializing side, you just need to provide the deserializers as configuration.

For example,

You can also set them at the binder level.

There is an optional property that you can set to force native decoding.

However, in the case of Kafka binder, this is unnecessary, as by the time it reaches the binder, Kafka already deserializes them using the configured deserializers.

By default, Kafka Streams binder always starts from the earliest offset for a new consumer. Sometimes, it is beneficial or required by the application to start from the latest offset. Kafka Streams binder allows you to do that.

Before we look at the solution, let us look at the following scenario.

We have a BiConsumer bean that requires two input bindings. In this case, the first binding is for a KStream and the second one is for a KTable. When running this application for the first time, by default, both bindings start from the earliest offset. What about I want to start from the latest offset due to some requirements? You can do this by enabling the following properties.

If you want only one binding to start from the latest offset and the other to consumer from the default earliest, then leave the latter binding out from the configuration.

Keep in mind that, once there are committed offsets, these setting are not honored and the committed offsets take precedence.

I have a Kafka producer application and I want to keep track of all my successful sendings.

Let us assume that we have this following supplier in the application.

Then, we need to define a new MessageChannel bean to capture all the successful send information.

Next, define this property in the application configuration to provide the bean name for the recordMetadataChannel.

At this point, successful sent information will be sent to the fooRecordChannel.

You can write an IntegrationFlow as below to see the information.

In the handle method, the payload is what got sent to Kafka and the message headers contain a special key called kafka_recordMetadata. Its value is a RecordMetadata that contains information about topic partition, current offset etc.

I have a Kafka producer application that sets some headers, but they are missing in the consumer application. Why is that?

Under normal circumstances, this should be fine.

Imagine, you have the following producer.

On the consumer side, you should still see the header "foo", and the following should not give you any issues.

If you provide a custom header mapper in the application, then this won’t work. Let’s say you have an empty KafkaHeaderMapper in the application.

If that is your implementation, then you will miss the foo header on the consumer. Chances are that, you may have some logic inside those KafkaHeaderMapper methods. You need the following to populate the foo header.

That will properly populate the foo header from the producer to consumer.

In Spring Cloud Stream, the id header is a special header, but some applications may want to have special custom id headers - something like custom-id or ID or Id. The first one (custom-id) will propagate without any custom header mapper from producer to consumer. However, if you produce with a variant of the framework reserved id header - such as ID, Id, iD etc. then you will run into issues with the internals of the framework. See this StackOverflow thread fore more context on this use case. In that case, you must use a custom KafkaHeaderMapper to map the case-sensitive id header. For example, let’s say you have the following producer.

The header Id above will be gone from the consuming side as it clashes with the framework id header. You can provide a custom KafkaHeaderMapper to solve this issue.

By doing this, both id and Id headers will be available from the producer to the consumer side.

How do I produce transactional messages to multiple Kafka topics?

For more context, see this StackOverflow question.

Use transactional support in Kafka binder for transactions and then provide an AfterRollbackProcessor. In order to produce to multiple topics, use StreamBridge API.

Below are the code snippets for this:

in order to test, you can use the following:

Some important notes:

Please ensure that you don’t have any DLQ settings on the application configuration as we manually configure DLT (By default it will be published to a topic named input.DLT based on the initial consumer function). Also, reset the maxAttempts on consumer binding to 1 in order to avoid retries by the binder. It will be max tried a total of 3 in the example above (initial try + the 2 attempts in the FixedBackoff).

See the StackOverflow thread for more details on how to test this code. If you are using Spring Cloud Stream to test it by adding more consumer functions, make sure to set the isolation-level on the consumer binding to read-committed.

This StackOverflow thread is also related to this discussion.

How can I run multiple instances of the pollable consumers and generate unique client.id for each instance?

Assuming that I have the following definition:

When running the application, the Kafka consumer generates a client.id (something like consumer-my-group-1). For each instance of the application that is running, this client.id will be the same, causing unexpected issues.

In order to fix this, you can add the following property on each instance of the application:

See this GitHub issue for more details.