Taps can be created at various producer endpoints in a stream. For a stream like this:

stream create --definition "http | step1: transform --expression=payload.toUpperCase() | step2: transform --expression=payload+'!' | log" --name mainstream --deploy

taps can be created at the output of http, step1 and step2.

To create a stream that acts as a 'tap' on another stream requires to specify the source destination name for the tap stream. The syntax for source destination name is:

`:<streamName>.<label/appName>`

To create a tap at the output of http in the stream above, the source destination name is mainstream.http To create a tap at the output of the first transform app in the stream above, the source destination name is mainstream.step1

The tap stream DSL looks like this:

stream create --definition ":mainstream.http > counter" --name tap_at_http --deploy

stream create --definition ":mainstream.step1 > jdbc" --name tap_at_step1_transformer --deploy

Note the colon (:) prefix before the destination names. The colon allows the parser to recognize this as a destination name instead of an app name.

When a stream is comprised of multiple apps with the same name, they must be qualified with labels:

stream create --definition "http | firstLabel: transform --expression=payload.toUpperCase() | secondLabel: transform --expression=payload+'!' | log" --name myStreamWithLabels --deploy

Instead of referencing a source or sink applications, you can use a named destination. A named destination corresponds to a specific destination name in the middleware broker (Rabbit, Kafka, etc.,). When using the | symbol, applications are connected to each other using messaging middleware destination names created by the Data Flow server. In keeping with the unix analogy, one can redirect standard input and output using the less-than < greater-than > charaters. To specify the name of the destination, prefix it with a colon :. For example the following stream has the destination name in the source position:

dataflow:>stream create --definition ":myDestination > log" --name ingest_from_broker --deploy

This stream receives messages from the destination myDestination located at the broker and connects it to the log app. You can also create additional streams that will consume data from the same named destination.

The following stream has the destination name in the sink position:

dataflow:>stream create --definition "http > :myDestination" --name ingest_to_broker --deploy

It is also possible to connect two different destinations (source and sink positions) at the broker in a stream.

dataflow:>stream create --definition ":destination1 > :destination2" --name bridge_destinations --deploy

In the above stream, both the destinations (destination1 and destination2) are located in the broker. The messages flow from the source destination to the sink destination via a bridge app that connects them.

Using named destinations, you can support Fan-in and Fan-out use cases. Fan-in use cases are when multiple sources all send data to the same named destination. For example

s3 > :data
ftp > :data
http > :data

Would direct the data payloads from the Amazon S3, FTP, and HTTP sources to the same named destination called data. Then an additional stream created with the DSL expression

:data > file

would have all the data from those three sources sent to the file sink.

The Fan-out use case is when you determine the destination of a stream based on some information that is only known at runtime. In this case, the Router Application can be used to specify how to direct the incoming message to one of N named destinations.