The following stream

dataflow:> stream create --definition "time | log" --name ticktock

can have application properties defined at the time of stream creation.

The shell command app info displays the white-listed application properties for the application. For more info on the property white listing refer to Section 19.1, “Whitelisting application properties”

Below are the white listed properties for the app time:

dataflow:> app info source:time
╔══════════════════════════════╤══════════════════════════════╤══════════════════════════════╤══════════════════════════════╗
║         Option Name          │         Description          │           Default            │             Type             ║
╠══════════════════════════════╪══════════════════════════════╪══════════════════════════════╪══════════════════════════════╣
║trigger.time-unit             │The TimeUnit to apply to delay│<none>                        │java.util.concurrent.TimeUnit ║
║                              │values.                       │                              │                              ║
║trigger.fixed-delay           │Fixed delay for periodic      │1                             │java.lang.Integer             ║
║                              │triggers.                     │                              │                              ║
║trigger.cron                  │Cron expression value for the │<none>                        │java.lang.String              ║
║                              │Cron Trigger.                 │                              │                              ║
║trigger.initial-delay         │Initial delay for periodic    │0                             │java.lang.Integer             ║
║                              │triggers.                     │                              │                              ║
║trigger.max-messages          │Maximum messages per poll, -1 │1                             │java.lang.Long                ║
║                              │means infinity.               │                              │                              ║
║trigger.date-format           │Format for the date value.    │<none>                        │java.lang.String              ║
╚══════════════════════════════╧══════════════════════════════╧══════════════════════════════╧══════════════════════════════╝

Below are the white listed properties for the app log:

dataflow:> app info sink:log
╔══════════════════════════════╤══════════════════════════════╤══════════════════════════════╤══════════════════════════════╗
║         Option Name          │         Description          │           Default            │             Type             ║
╠══════════════════════════════╪══════════════════════════════╪══════════════════════════════╪══════════════════════════════╣
║log.name                      │The name of the logger to use.│<none>                        │java.lang.String              ║
║log.level                     │The level at which to log     │<none>                        │org.springframework.integratio║
║                              │messages.                     │                              │n.handler.LoggingHandler$Level║
║log.expression                │A SpEL expression (against the│payload                       │java.lang.String              ║
║                              │incoming message) to evaluate │                              │                              ║
║                              │as the logged message.        │                              │                              ║
╚══════════════════════════════╧══════════════════════════════╧══════════════════════════════╧══════════════════════════════╝

The application properties for the time and log apps can be specified at the time of stream creation as follows:

dataflow:> stream create --definition "time --fixed-delay=5 | log --level=WARN" --name ticktock

Note that the properties fixed-delay and level defined above for the apps time and log are the 'short-form' property names provided by the shell completion. These 'short-form' property names are applicable only for the white-listed properties and in all other cases, only fully qualified property names should be used.

If you would like to have multiple instances of an application in the stream, you can include a property with the deploy command:

dataflow:> stream deploy --name ticktock --properties "app.time.count=3"

Note that count is the reserved property name used by the underlying deployer. Hence, if the application also has a custom property named count, it is not supported when specified in 'short-form' form during stream deployment as it could conflict with the instance count deployer property. Instead, the count as a custom application property can be specified in its fully qualified form (example: app.foo.bar.count) during stream deployment or it can be specified using 'short-form' or fully qualified form during the stream creation where it will be considered as an app property.

	Important
	See Chapter 27, Using Labels in a Stream.

When using the Spring Cloud Dataflow Shell, there are two ways to provide deployment properties: either inline or via a file reference. Those two ways are exclusive and documented below:

stream deploy foo
    --properties "app.transform.count=2,app.transform.producer.partitionKeyExpression=payload"

stream deploy foo --propertiesFile myprops.properties

where myprops.properties contains:

app.transform.count=2
app.transform.producer.partitionKeyExpression=payload

Both the above properties will be passed as deployment properties for the stream foo above.

The application properties can also be specified when deploying a stream. When specified during deployment, these application properties can either be specified as 'short-form' property names (applicable for white-listed properties) or fully qualified property names. The application properties should have the prefix "app.<appName/label>".

For example, the stream

dataflow:> stream create --definition "time | log" --name ticktock

can be deployed with application properties using the 'short-form' property names:

dataflow:>stream deploy ticktock --properties "app.time.fixed-delay=5,app.log.level=ERROR"

When using the app label,

stream create ticktock --definition "a: time | b: log"

the application properties can be defined as:

stream deploy ticktock --properties "app.a.fixed-delay=4,app.b.level=ERROR"

Spring Cloud Data Flow sets the required Spring Cloud Stream properties for the applications inside the stream. Most importantly, the spring.cloud.stream.bindings.<input/output>.destination is set internally for the apps to bind.

If someone wants to override any of the Spring Cloud Stream properties, they can be set via deployment properties.

For example, for the below stream

dataflow:> stream create --definition "http | transform --expression=payload.getValue('hello').toUpperCase() | log" --name ticktock

if there are multiple binders available in the classpath for each of the applications and the binder is chosen for each deployment then the stream can be deployed with the specific Spring Cloud Stream properties as:

dataflow:>stream deploy ticktock --properties "app.time.spring.cloud.stream.bindings.output.binder=kafka,app.transform.spring.cloud.stream.bindings.input.binder=kafka,app.transform.spring.cloud.stream.bindings.output.binder=rabbit,app.log.spring.cloud.stream.bindings.input.binder=rabbit"

	Note
	Overriding the destination names is not recommended as Spring Cloud Data Flow takes care of setting this internally.

A Spring Cloud Stream application can have producer and consumer properties set per-binding basis. While Spring Cloud Data Flow supports specifying short-hand notation for per binding producer properties such as partitionKeyExpression, partitionKeyExtractorClass as described in Section 20.2.6, “Passing stream partition properties during stream deployment”, all the supported Spring Cloud Stream producer/consumer properties can be set as Spring Cloud Stream properties for the app directly as well.

The consumer properties can be set for the inbound channel name with the prefix app.[app/label name].spring.cloud.stream.bindings.<channelName>.consumer. and the producer properties can be set for the outbound channel name with the prefix app.[app/label name].spring.cloud.stream.bindings.<channelName>.producer.. For example, the stream

dataflow:> stream create --definition "time | log" --name ticktock

can be deployed with producer/consumer properties as:

dataflow:>stream deploy ticktock --properties "app.time.spring.cloud.stream.bindings.output.producer.requiredGroups=myGroup,app.time.spring.cloud.stream.bindings.output.producer.headerMode=raw,app.log.spring.cloud.stream.bindings.input.consumer.concurrency=3,app.log.spring.cloud.stream.bindings.input.consumer.maxAttempts=5"

The binder specific producer/consumer properties can also be specified in a similar way.

For instance

dataflow:>stream deploy ticktock --properties "app.time.spring.cloud.stream.rabbit.bindings.output.producer.autoBindDlq=true,app.log.spring.cloud.stream.rabbit.bindings.input.consumer.transacted=true"

A common pattern in stream processing is to partition the data as it is streamed. This entails deploying multiple instances of a message consuming app and using content-based routing so that messages with a given key (as determined at runtime) are always routed to the same app instance. You can pass the partition properties during stream deployment to declaratively configure a partitioning strategy to route each message to a specific consumer instance.

See below for examples of deploying partitioned streams:

In summary, an app is partitioned if its count is > 1 and the previous app has a partitionKeyExtractorClass or partitionKeyExpression (class takes precedence). When a partition key is extracted, the partitioned app instance is determined by invoking the partitionSelectorClass, if present, or the partitionSelectorExpression % partitionCount, where partitionCount is application count in the case of RabbitMQ, and the underlying partition count of the topic in the case of Kafka.

If neither a partitionSelectorClass nor a partitionSelectorExpression is present the result is key.hashCode() % partitionCount.

In a stream definition you can specify that the input or the output of an application need to be converted to a different type. You can use the inputType and outputType properties to specify the content type for the incoming data and outgoing data, respectively.

For example, consider the following stream:

dataflow:>stream create tuple --definition "http | filter --inputType=application/x-spring-tuple
 --expression=payload.hasFieldName('hello') | transform --expression=payload.getValue('hello').toUpperCase()
 | log" --deploy

The http app is expected to send the data in JSON and the filter app receives the JSON data and processes it as a Spring Tuple. In order to do so, we use the inputType property on the filter app to convert the data into the expected Spring Tuple format. The transform application processes the Tuple data and sends the processed data to the downstream log application.

When sending some data to the http application:

dataflow:>http post --data {"hello":"world","foo":"bar"} --contentType application/json --target http://localhost:<http-port>

At the log application you see the content as follows:

INFO 18745 --- [transform.tuple-1] log.sink                                 : WORLD

Depending on how applications are chained, the content type conversion can be specified either as via the --outputType in the upstream app or as an --inputType in the downstream app. For instance, in the above stream, instead of specifying the --inputType on the 'transform' application to convert, the option --outputType=application/x-spring-tuple can also be specified on the 'http' application.

For the complete list of message conversion and message converters, please refer Spring Cloud Stream documentation.

Application properties that are defined during deployment override the same properties defined during the stream creation.

For example, the following stream has application properties defined during stream creation:

dataflow:> stream create --definition "time --fixed-delay=5 | log --level=WARN" --name ticktock

To override these application properties, one can specify the new property values during deployment:

dataflow:>stream deploy ticktock --properties "app.time.fixed-delay=4,app.log.level=ERROR"

If you would like to have multiple instances of an application in the stream, you can include a property with the deploy command:

dataflow:> stream deploy --name ticktock --properties "app.time.count=3"

Note that count is the reserved property name used by the underlying deployer. Hence, if the application also has a custom property named count, it is not supported when specified in 'short-form' form during stream deployment as it could conflict with the instance count deployer property. Instead, the count as a custom application property can be specified in its fully qualified form (example: app.foo.bar.count) during stream deployment or it can be specified using 'short-form' or fully qualified form during the stream creation where it will be considered as an app property.

	Important
	See Chapter 27, Using Labels in a Stream.

When using the Spring Cloud Dataflow Shell, there are two ways to provide deployment properties: either inline or via a file reference. Those two ways are exclusive and documented below:

stream deploy foo
    --properties "app.transform.count=2,app.transform.producer.partitionKeyExpression=payload"

stream deploy foo --propertiesFile myprops.properties

where myprops.properties contains:

app.transform.count=2
app.transform.producer.partitionKeyExpression=payload

Both the above properties will be passed as deployment properties for the stream foo above.

In case of using YAML as the format for the deployment properties, use the .yaml or .yml file extention when deploying the stream,

stream deploy foo --propertiesFile myprops.yaml

where myprops.yaml contains:

app:
  transform:
    count: 2
    producer:
      partitionKeyExpression: payload