Spring Cloud Stream Application Starters provide you with predefined Spring Cloud Stream applications that you can run independently or with Spring Cloud Data Flow. You can also use the starters as a basis for creating your own applications. They include:
You can find a detailed listing of all the starters and as their options in the corresponding section of this guide.
You can find all available app starter repositories in this GitHub Organization.
As a user of Spring Cloud Stream Application Starters you have access to two types of artifacts.
Starters are libraries that contain the complete configuration of a Spring Cloud Stream application with a specific role (e.g. an HTTP source that receives HTTP POST requests and forwards the data on its output channel to downstream Spring Cloud Stream applications). Starters are not executable applications, and are intended to be included in other Spring Boot applications, along with a Binder implementation.
Prebuilt applications are Spring Boot applications that include the starters and a Binder implementation. Prebuilt applications are uberjars and include minimal code required to execute standalone. For each starter, the project provides a prebuilt version including the Kafka Binder (one each for 0.9 and 0.10 versions of Kafka) and a prebuilt version including the Rabbit MQ Binder.
![]() | Note |
---|---|
Only starters are present in the source code of the project. Prebuilt applications are generated according to the stream apps generator maven plugin. |
Based on their target application type, starters can be either:
You can easily identify the type and functionality of a starter based on its name.
All starters are named following the convention spring-cloud-starter-stream-<type>-<functionality>
.
For example spring-cloud-starter-stream-source-file
is a starter for a file source that polls a directory and sends file data on the output channel (read the reference documentation of the source for details).
Conversely, spring-cloud-starter-stream-sink-cassandra
is a starter for a Cassandra sink that writes the data that it receives on the input channel to Cassandra (read the reference documentation of the sink for details).
The prebuilt applications follow a naming convention too: <functionality>-<type>-<binder>
. For example, cassandra-sink-kafka-10
is a Cassandra sink using the Kafka binder that is running with Kafka version 0.10.
You either get access to the artifacts produced by Spring Cloud Stream Application Starters via Maven, Docker, or building the artifacts yourself.
Starters are available as Maven artifacts in the Spring repositories. You can add them as dependencies to your application, as follows:
<dependency> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>spring-cloud-starter-stream-sink-cassandra</artifactId> <version>1.0.0.BUILD-SNAPSHOT</version> </dependency>
From this, you can infer the coordinates for other starters found in this guide.
While the version may vary, the group will always remain org.springframework.cloud.stream.app
and the artifact id follows the naming convention spring-cloud-starter-stream-<type>-<functionality>
described previously.
Prebuilt applications are available as Maven artifacts too.
It is not encouraged to use them directly as dependencies, as starters should be used instead.
Following the typical Maven <group>:<artifactId>:<version>
convention, they can be referenced for example as:
org.springframework.cloud.stream.app:cassandra-sink-rabbit:1.0.0.BUILD-SNAPSHOT
Just as with the starters, you can infer the coordinates for other prebuilt applications found in the guide.
The group will be always org.springframework.cloud.stream.app
.
The version may vary.
The artifact id follows the format <functionality>-<type>-<binder>
previously described.
You can download the executable jar artifacts from the Spring Maven repositories. The root directory of the Maven repository that hosts release versions is repo.spring.io/release/org/springframework/cloud/stream/app/. From there you can navigate to the latest release version of a specific app, for example log-sink-rabbit-1.1.1.RELEASE.jar. Use the Milestone and Snapshot repository locations for Milestone and Snapshot executuable jar artifacts.
The Docker versions of the applications are available in Docker Hub, at hub.docker.com/r/springcloudstream/
. Naming and versioning follows the same general conventions as Maven, e.g.
docker pull springcloudstream/cassandra-sink-kafka-10
will pull the latest Docker image of the Cassandra sink with the Kafka binder that is running with Kafka version 0.10.
You can also build the project and generate the artifacts (including the prebuilt applications) on your own. This is useful if you want to deploy the artifacts locally or add additional features.
First, you need to generate the prebuilt applications. This is done by running the application generation Maven plugin. You can do so by simply invoking the maven build with the generateApps profile and install lifecycle.
mvn clean install -PgenerateApps
Each of the prebuilt applciation will contain:
pom.xml
file with the required dependencies (starter and binder)main
method of the application and imports the predefined configurationFor example, spring-cloud-starter-stream-sink-cassandra
will generate cassandra-sink-rabbit
, cassandra-sink-kafka-09
and cassandra-sink-kafka-10
as completely functional applications.
Apart from accessing the sources, sinks and processors already provided by the project, in this section we will describe how to:
Prebuilt applications are provided for both kafka and rabbit binders. But if you want to connect to a different middleware system, and you have a binder for it, you will need to create new artifacts.
<dependencies> <!- other dependencies --> <dependency> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>spring-cloud-starter-stream-sink-cassandra</artifactId> <version>1.0.0.BUILD-SNAPSHOT</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream-binder-gemfire</artifactId> <version>1.0.0.BUILD-SNAPSHOT</version> </dependency> </dependencies>
The next step is to create the project’s main class and import the configuration provided by the starter.
package org.springframework.cloud.stream.app.cassandra.sink.rabbit; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.stream.app.cassandra.sink.CassandraSinkConfiguration; import org.springframework.context.annotation.Import; @SpringBootApplication @Import(CassandraSinkConfiguration.class) public class CassandraSinkGemfireApplication { public static void main(String[] args) { SpringApplication.run(CassandraSinkGemfireApplication.class, args); } }
Spring Cloud Stream Application consists of regular Spring Boot applications with some additional conventions that facilitate generating prebuilt applications with the preconfigured binders. Sometimes, your solution may require additional applications that are not in the scope of out of the box Spring Cloud Stream Application Starters, or require additional tweaks and enhancements. In this section we will show you how to create custom applications that can be part of your solution, along with Spring Cloud Stream application starters. You have the following options:
If you want to add your own custom applications to your solution, you can simply create a new Spring Cloud Stream app project with the binder of your choice and run it the same way as the applications provided by Spring Cloud Stream Application Starters, independently or via Spring Cloud Data Flow. The process is described in the Getting Started Guide of Spring Cloud Stream.
An alternative way to bootstrap your application is to go to the Spring Initializr and choose a Spring Cloud Stream Binder of your choice. This way you already have the necessary infrastructure ready to go and mainly focus on the specifics of the application.
The following requirements need to be followed when you go with this option:
input
for sources - the simplest way to do so is by using the predefined interface org.spring.cloud.stream.messaging.Source
;output
for sinks - the simplest way to do so is by using the predefined interface org.spring.cloud.stream.messaging.Sink
;input
and an outbound channel named output
for processors - the simplest way to do so is by using the predefined interface org.spring.cloud.stream.messaging.Processor
.You can also reuse the starters provided by Spring Cloud Stream Application Starters to create custom components, enriching the behavior of the application.
For example, you can add a Spring Security layer to your HTTP source, add additional configurations to the ObjectMapper
used for JSON transformation wherever that happens, or change the JDBC driver or Hadoop distribution that the application is using.
In order to do this, you should set up your project following a process similar to customizing a binder.
In fact, customizing the binder is the simplest form of creating a custom component.
As a reminder, this involves:
After doing so, you can simply add the additional configuration for the extra features of your application.
If you’re looking to patch the pre-built applications to accommodate addition of new dependencies, you can use the following example as the reference. Let’s review the steps to add mysql
driver to jdbc-sink
application.
mysql
java-driver dependency<dependencies> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.37</version> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream-binder-rabbit</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>spring-cloud-starter-stream-sink-jdbc</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies>
jdbc
sink, it is: @Import(org.springframework.cloud.stream.app.jdbc.sink.JdbcSinkConfiguration.class)
. You can find the configuration class for other applications in their respective repositories.@SpringBootApplication @Import(org.springframework.cloud.stream.app.jdbc.sink.JdbcSinkConfiguration.class) public class DemoApplication { public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } }
jdbc-sink
application now includes mysql
driver in itIn this section, we will explain how to develop a custom source/sink/processor application and then generate maven and docker artifacts for it with the necessary middleware bindings using the existing tooling provided by the spring cloud stream app starter infrastructure. For explanation purposes, we will assume that we are creating a new source application for a technology named foobar.
app-starters-build
Please follow the instructions above for designing a proper Spring Cloud Stream Source. You may also look into the existing
starters for how to structure a new one. The default naming for the main @Configuration
class is
FoobarSourceConfiguration
and the default package for this @Configuration
is org.springfamework.cloud.stream.app.foobar.source
. If you have a different class/package name, see below for
overriding that in the app generator. The technology/functionality name for which you create
a starter can be a hyphenated stream of strings such as in scriptable-transform
which is a processor type in the
module spring-cloud-starter-stream-processor-scriptable-transform
.
The starters in spring-cloud-stream-app-starters
are slightly different from the other starters in spring-boot and
spring-cloud in that here we don’t provide a way to auto configure any configuration through spring factories mechanism.
Rather, we delegate this responsibility to the maven plugin that is generating the binder based apps. Therefore, you don’t
have to provide a spring.factories file that lists all your configuration classes.
foobar-app-starters-build
)build
section. This will add the necessary plugin configuration for app generation as well as generating proper documentation metadata.
Please ensure that your root pom inherits app-starters-build as the base configuration for the plugins is specified there.<build> <plugins> <plugin> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-app-starter-doc-maven-plugin</artifactId> </plugin> <plugin> <groupId>org.springframework.cloud.stream.app.plugin</groupId> <artifactId>spring-cloud-stream-app-maven-plugin</artifactId> <configuration> <generatedProjectHome>${session.executionRootDirectory}/apps</generatedProjectHome> <generatedProjectVersion>${project.version}</generatedProjectVersion> <bom> <name>scs-bom</name> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>foobar-app-dependencies</artifactId> <version>${project.version}</version> </bom> <generatedApps> <foobar-source/> </generatedApps> </configuration> </plugin> </plugins> </build>
More information about the maven plugin used above to generate the apps can be found here: github.com/spring-cloud/spring-cloud-stream-app-maven-plugin
If you did not follow the default convention expected by the plugin for where it is looking for the main configuration
class, which is org.springfamework.cloud.stream.app.foobar.source.FoobarSourceConfiguration
, you can override that in
the configuration for the plugin. For example, if your main configuration class is foo.bar.SpecialFooBarConfiguration.class
,
this is how you can tell the plugin to override the default.
<foobar-source> <autoConfigClass>foo.bar.SpecialFooBarConfiguration.class</autoConfigClass> </foobar-source>
foobar-app-dependencies
). This is the bom (bill of material) for this project. It is advised that this bom is inherited from spring-cloud-dependencies-parent
. Please see other starter repositories for guidelines.<dependencyManagement> ... ... <dependency> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>spring-cloud-starter-stream-source-foobar</artifactId> <version>1.0.0.BUILD-SNAPSHOT</version> </dependency> ... ...
./mvnw clean install -PgenerateApps
This will generate the binder based foobar source apps in a directory named apps
at the root of the repository.
If you want to change the location where the apps are generated, for instance `/tmp/scs-apps
, you can do it in the
configuration section of the plugin.
<configuration> ... <generatedProjectHome>/tmp/scs-apps</generatedProjectHome> ... </configuration
By default, we generate apps for both Kafka 09/10 and Rabbitmq binders - spring-cloud-stream-binder-kafka
and
spring-cloud-stream-binder-rabbit
. Say, if you have a custom binder you created for some middleware (say JMS),
which you need to generate apps for foobar source, you can add that binder to the binders list in the configuration
section as in the following.
<binders> <jms /> </binders>
Please note that this would only work, as long as there is a binder with the maven coordinates of
org.springframework.cloud.stream
as group id and spring-cloud-stream-binder-jms
as artifact id.
This artifact needs to be specified in the BOM above and available through a maven repository as well.
If you have an artifact that is only available through a private internal maven repository (may be an enterprise wide Nexus repo that you use globally across teams), and you need that for your app, you can define that as part of the maven plugin configuration.
For example,
<configuration> ... <extraRepositories> <repository> <id>private-internal-nexus</id> <url>.../</url> <name>...</name> <snapshotEnabled>...</snapshotEnabled> </repository> </extraRepositories> </configuration>
Then you can define this as part of your app tag:
<foobar-source> <extraRepositories> <private-internal-nexus /> </extraRepositories> </foobar-source>
apps
at the root of the repository by default, unless you changed
it elsewhere as described above).Here you will see foobar-source-kafka-09
, foobar-source-kafka-10
and foobar-source-rabbit
.
If you added more binders as described above, you would see that app as well here - for example foobar-source-jms.
You can import these apps directly into your IDE of choice if you further want to do any customizations on them. Each of them is a self contained spring boot application project.
For the generated apps, the parent is spring-boot-starter-parent
as required by the underlying Spring Initializr library.
You can cd into these custom foobar-source directories and do the following to build the apps:
cd foo-source-kafka-10
mvn clean install
This would install the foo-source-kafka-10 into your local maven cache (~/.m2 by default).
The app generation phase adds an integration test to the app project that is making sure that all the spring
components and contexts are loaded properly. However, these tests are not run by default when you do a mvn install
.
You can force the running of these tests by doing the following:
mvn clean install -DskipTests=false
One important note about running these tests in generated apps:
If your application’s spring beans need to interact with
some real services out there or expect some properties to be present in the context, these tests will fail unless you make
those things available. An example would be a Twitter Source, where the underlying spring beans are trying to create a
twitter template and will fail if it can’t find the credentials available through properties. One way to solve this and
still run the generated context load tests would be to create a mock class that provides these properties or mock beans
(for example, a mock twitter template) and tell the maven plugin about its existence. You can use the existing module
app-starters-test-support
for this purpose and add the mock class there.
See the class org.springframework.cloud.stream.app.test.twitter.TwitterTestConfiguration
for reference.
You can create a similar class for your foobar source - FoobarTestConfiguration
and add that to the plugin configuration.
You only need to do this if you run into this particular issue of spring beans are not created properly in the
integration test in the generated apps.
<foobar-source> <extraTestConfigClass>org.springframework.cloud.stream.app.test.foobar.FoobarTestConfiguration.class</extraTestConfigClass> </foobar-source>
When you do the above, this test configuration will be automatically imported into the context of your test class.
Also note that, you need to regenerate the apps each time you make a configuration change in the plugin.
target
directories of the respective apps and also as
maven artifacts in your local maven repository. Go to the target
directory and run the following:java -jar foobar-source-kafa-10.jar
[Ensure that you have kafka running locally when you do this]
It should start the application up.
mvn clean package docker:build
This creates the docker image under the target/docker/springcloudstream
directory. Please ensure that the Docker
container is up and running and DOCKER_HOST environment variable is properly set before you try docker:build
.
All the generated apps from the various app repositories are uploaded to Docker Hub
However, for a custom app that you build, this won’t be uploaded to docker hub under springcloudstream
repository.
If you think that there is a general need for this app, you should try contributing this starter as a new repository to Spring Cloud Stream App Starters.
Upon review, this app then can be eventually available through the above location in docker hub.
If you still need to push this to docker hub under a different repository (may be an enterprise repo that you manage for your organization) you can take the following steps.
Go to the pom.xml of the generated app [ example - foo-source-kafka/pom.xml
]
Search for springcloudstream
. Replace with your repository name.
Then do this:
mvn clean package docker:build docker:push -Ddocker.username=[provide your username] -Ddocker.password=[provide password]
This would upload the docker image to the docker hub in your custom repository.
In the following sections, you can find a brief faq on various things that we discussed above and a few other infrastructure related topics.
What is the parent for stream app starters?
The parent for all app starters is app-starters-build
which is coming from the core project. github.com/spring-cloud-stream-app-starters/core
For example:
<parent> <groupId>org.springframework.cloud.stream.app</groupId> <artifactId>app-starters-build</artifactId> <version>1.3.0.RELEASE</version> <relativePath/> </parent>
app-starters-core-dependencies
.
We need this bom during app generation to pull down all the core dependencies.app-starters-build
artfiact. This same BOM is referenced through the maven plugin configuration for the app generation.
The generated apps thus will include this bom also in their pom.xml files.What spring cloud stream artifacts does the parent artifact (app-starters-build
) include?
What other artfiacts are available through the parent app-starters-build
and where are they coming from?
In addition to the above artifacts, the artifacts below also included in app-starters-build
by default.
Can you summarize all the BOM’s that SCSt app starters depend on? All SCSt app starters have access to dependencies defined in the following BOM’s and other dependencies from any other BOM’s these three boms import transitively as in the case of Spring Integration:
app-starter-build
as the parent which in turn has spring-cloud-build
as parent. The above documentation states that the
generated apps have spring-boot-starter
as the parent. Why the mismatch?
There is no mismatch per se, but a slight subtlety. As the question frames, each app starter has access to artifacts managed all the way through spring-cloud-build
at compile time.
However, this is not the case for the generated apps at runtime. Generated apps are managed by boot. Their parent is spring-boot-starter
that imports spring-boot-dependencies
bom that includes a majority of the components that these apps need.
The additional dependencies that the generated application needs are managed by including a BOM specific to each application starter.time-app-dependencies
.
This is an important BOM. At runtime, the generated apps get the versions used in their dependencies through a BOM that is managing the dependencies. Since all the boms
that we specified above only for the helper artifacts, we need a place to manage the starters themselves. This is where the app specific BOM comes into play.
In addition to this need, as it becomes clear below, there are other uses for this BOM such as dependency overrides etc. But in a nutshell, all the starter dependencies go to this BOM.
For instance, take TCP repo as an example. It has a starter for source, sink, client processor etc. All these dependencies are managed through the app specific tcp-app-dependencies
bom.
This bom is provided to the app generator maven plugin in addition to the core bom. This app specific bom has spring-cloud-dependencies-parent
as parent.spring-cloud-stream-app-starters
organization where you can start contributing the starters and other components.How do I override Spring Integration version that is coming from spring-boot-dependencies by default? The following solution only works if the versions you want to override are available through a new Spring Integration BOM. Go to your app starter specific bom. Override the property as following:
<spring-integration.version>VERSION GOES HERE</spring-integration.version>
Then add the following in the dependencies management section in the BOM.
<dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-bom</artifactId> <version>${spring-integration.version}</version> <scope>import</scope> <type>pom</type> </dependency>
How do I override spring-cloud-stream artifacts coming by default in spring-cloud-dependencies defined in core BOM? The following solution only works if the versions you want to override are available through a new Spring-Cloud-Dependencies BOM. Go to your app starter specific bom. Override the property as following:
<spring-cloud-dependencies.version>VERSION GOES HERE</spring-cloud-dependencies.version>
Then add the following in the dependencies management section in the BOM.
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>${spring-cloud-dependencies.version}</version> <scope>import</scope> <type>pom</type> </dependency>
What if there is no spring-cloud-dependencies BOM available that contains my versions of spring-cloud-stream, but there is a spring-cloud-stream BOM available? Go to your app starter specific BOM. Override the property as below.
<spring-cloud-stream.version>VERSION GOES HERE</spring-cloud-stream.version>
Then add the following in the dependencies management section in the BOM.
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream-dependencies</artifactId> <version>${spring-cloud-stream.version}</version> <scope>import</scope> <type>pom</type> </dependency>
What if I want to override a single artifact that is provided through a bom? For example spring-integration-java-dsl? Go to your app starter BOM and add the following property with the version you want to override:
<spring-integration-java-dsl.version>VERSION GOES HERE</spring-integration-java-dsl.version>
Then in the dependency management section add the following:
<dependency> <groupId>org.springframework.integration</groupId> <artifactId>spring-integration-java-dsl</artifactId> <version>${spring-integration-java-dsl.version}</version> </dependency>
How do I override the boot version used in a particular app? When you generate the app, override the boot version as follows.
./mvnw clean install -PgenerateApps -DbootVersion=<boot version to override>
For example: ./mvnw clean install -PgenerateApps -DbootVersion=2.0.0.BUILD-SNAPSHOT
You can also override the boot version more permanently by overriding the following property in your starter pom.
<bootVersion>2.0.0.BUILD-SNAPSHOT</bootVersion>