2. Java DSL

./mvnw clean package

With no command line options, the application will deploy the stream http --server.port=9900 | splitter --expression=payload.split(' ') | log using the URI localhost:9393 to connect to the Data Flow server. There is also a command line option --style whose value can be either definition or fluent. This options picks which JavaDSL style will execute. Both are identical in terms of behavior. The spring-cloud-dataflow-rest-client project provides auto-configuration for DataFlowOperations and StreamBuilder

The properties in DataFlowClientProperties can be used to configure the connection to the Data Flow server. The common property to start using is spring.cloud.dataflow.client.uri

@Autowired
private DataFlowOperations dataFlowOperations;

@Autowired
private StreamBuilder builder;

You can use those beans to build streams as well as work directly with `DataFlowOperations" REST client.

The definition style has code of the style

Stream woodchuck = builder
        .name("woodchuck")
        .definition("http --server.port=9900 | splitter --expression=payload.split(' ') | log")
        .create()
        .deploy(deploymentProperties);

while the fluent style has code of the style

Stream woodchuck = builder.name("woodchuck")
        .source(source)
        .processor(processor)
        .sink(sink)
        .create()
        .deploy(deploymentProperties);

where source, processor, and sink variables were defined as @Bean`s of the type `StreamApplication

@Bean
public StreamApplication source() {
  return new StreamApplication("http").addProperty("server.port", 9900);
}

Another useful class is the DeploymentPropertiesBuilder which aids in the creation of the Map of properties required to deploy stream applications.

	private Map<String, String> createDeploymentProperties() {
		DeploymentPropertiesBuilder propertiesBuilder = new DeploymentPropertiesBuilder();
		propertiesBuilder.memory("log", 512);
		propertiesBuilder.count("log",2);
		propertiesBuilder.put("app.splitter.producer.partitionKeyExpression", "payload");
		return propertiesBuilder.build();
	}

This sample demonstrates the use of the local Data Flow Server, but you can pass in the option --uri to point to another Data Flow server instance that is running elsewhere.

$ java -jar target/scdfdsl-0.0.1-SNAPSHOT.jar

You will then see the following output.

Deploying stream.
Wating for deployment of stream.
Wating for deployment of stream.
Wating for deployment of stream.
Wating for deployment of stream.
Wating for deployment of stream.
Letting the stream run for 2 minutes.

To verify that the application has been deployed successfully, will tail the logs of one of the log sinks and post some data to the http source. You can find the location for the logs of one of the log sink applications by looking in the Data Flow server’s log file.

curl http://localhost:9900 -H "Content-Type:text/plain"   -X POST -d "how much wood would a woodchuck chuck if a woodchuck could chuck wood"

Tailing the log file of the first instance

cd /tmp/spring-cloud-dataflow-4323595028663837160/woodchuck-1511390696355/woodchuck.log
tail -f stdout_0.log

2017-11-22 18:04:08.631  INFO 26652 --- [r.woodchuck-0-1] log-sink          : how
2017-11-22 18:04:08.632  INFO 26652 --- [r.woodchuck-0-1] log-sink          : chuck
2017-11-22 18:04:08.634  INFO 26652 --- [r.woodchuck-0-1] log-sink          : chuck

Tailing the log file of the second instance

cd /tmp/spring-cloud-dataflow-4323595028663837160/woodchuck-1511390696355/woodchuck.log
tail -f stdout_1.log

You should see the output

$ tail -f stdout_1.log
2017-11-22 18:04:08.636  INFO 26655 --- [r.woodchuck-1-1] log-sink          : much
2017-11-22 18:04:08.638  INFO 26655 --- [r.woodchuck-1-1] log-sink          : wood
2017-11-22 18:04:08.639  INFO 26655 --- [r.woodchuck-1-1] log-sink          : would
2017-11-22 18:04:08.640  INFO 26655 --- [r.woodchuck-1-1] log-sink          : a
2017-11-22 18:04:08.641  INFO 26655 --- [r.woodchuck-1-1] log-sink          : woodchuck
2017-11-22 18:04:08.642  INFO 26655 --- [r.woodchuck-1-1] log-sink          : if
2017-11-22 18:04:08.644  INFO 26655 --- [r.woodchuck-1-1] log-sink          : a
2017-11-22 18:04:08.645  INFO 26655 --- [r.woodchuck-1-1] log-sink          : woodchuck
2017-11-22 18:04:08.646  INFO 26655 --- [r.woodchuck-1-1] log-sink          : could
2017-11-22 18:04:08.647  INFO 26655 --- [r.woodchuck-1-1] log-sink          : wood

Note that the partitioning is done based on the hash of the java.lang.String object.

Optionally, if you have enabled authentication in SCDF, there are three different ways to authorize the sample application (i.e. the client).

Use basic authentication:

$ java -jar target/scdfdsl-0.0.1-SNAPSHOT.jar \
  --spring.cloud.dataflow.client.authentication.basic.username=user \
  --spring.cloud.dataflow.client.authentication.basic.password=password

Use OAuth client settings (UAA is used as the identity provider in this sample):

$ java -jar target/scdfdsl-0.0.1-SNAPSHOT.jar \
  --spring.cloud.dataflow.client.authentication.client-id=dataflow \
  --spring.cloud.dataflow.client.authentication.client-secret=secret \
  --spring.cloud.dataflow.client.authentication.token-uri=http://localhost:8080/uaa/oauth/token \
  --spring.cloud.dataflow.client.authentication.scope=dataflow.create,dataflow.deploy,dataflow.destroy,dataflow.manage,dataflow.modify,dataflow.schedule,dataflow.view

Use OAuth access token:

$ java -jar target/scdfdsl-0.0.1-SNAPSHOT.jar \
  --spring.cloud.dataflow.client.authentication.access-token=849228ed663e450ab5051c998eb71a4a

For example, if you’re using UAA as the identity provider backend, the access token can be requested with the following command:

$ curl 'http://localhost:8080/uaa/oauth/token' -i -X POST \
    -H 'Content-Type: application/x-www-form-urlencoded' \
    -H 'Accept: application/json' \
    -d 'client_id=dataflow&client_secret=secret&grant_type=password&username=user&password=password&token_format=opaque'

{
  "access_token":"849228ed663e450ab5051c998eb71a4a",
  ...
}