Deploy a Mesos and Marathon cluster.

The Mesosphere getting started guide provides a number of options for you to deploy a cluster. Many of the options listed there need some additional work to get going. For example, many Vagrant provisioned VMs are using deprecated versions of the Docker client. One of the most straightforward ways to setup a small cluster on your local development machine is to follow Exercises 1-4 and 12 from the Advanced Mesos Course. Note that in Exercise 12, it is preferable to install docker from the official docker instructions and use only the mesos configuraiton steps listed in Exercise 12. Minimesos also looks to be useful. For those wanted to setup a distributed cluster quickly, there is also an option to spin up a cluster on AWS using Mesosphere’s Datacenter Operation System on Amazon Web Services.

The rest of this getting started guide assumes that you have a working Mesos and Marathon cluster and know the Marathon endpoint URL.

Create a Redis service on the Mesos cluster.

The redis service will be used for messaging between modules in the stream. There is a sample application JSON file for redis in the spring-cloud-dataflow-admin-mesos repository that you can use as a starting point. The service discovery mechanism of the deployed modules depends on finding an application with the name redis. Depending on how large your cluster is, you way want to tweek the CPU and/or memory values.

Using the above JSON file and an Mesos and Marathon cluster installed you can deploy a Redis application instance by issuing the following command

curl -X POST http://192.168.33.10:8080/v2/apps -d @redis.json -H "Content-type: application/json"

Using the Marathon and Mesos UIs you can verify that redis is running on the cluster. Note the @ symbol to reference a file and that the Marathon endpoint URL is 192.168.33.10:8080 based on the configuration used in the Advanced Mesos Course instructions.

Run a local redis-server.

$ redis-sever

This is used by the locally running admin server to store the state of available module versions for stream definitions.

	Note
	If you are switching between milestone and snapshot versions of the admin server, flush the redis keys that contain the module’s version information. Calling flushdb is indiscriminate but a handy way to start from a clean redis state.

Download and run the Spring Cloud Data Flow Admin server for Mesos and Marathon.

$ wget http://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-dataflow-admin-mesos/1.0.0.BUILD-SNAPSHOT/spring-cloud-dataflow-admin-mesos-1.0.0.BUILD-SNAPSHOT.jar

$ java -jar spring-cloud-dataflow-admin-mesos-1.0.0.BUILD-SNAPSHOT.jar --marathon.apiEndpoint=http://192.168.33.10:8080 --remoteRepositories=https://repo.spring.io/libs-milestone

You can also pass in properties to set default values for memory and cpu resource request. For example --marathon.cpu=0.25 will by default allocate 25% of the CPU for the application vs. the default value of 50%. You can see all the available options in the MarathonProperties.java file. For the remoteRepositories property, you can provide a command deliminted list of values. For example, if you want to expose modules that are published to your local maven repository you can run a small local web server to expose Maven’s local repository directory

cd ~/.m2/repository
python -m SimpleHTTPServer 8000

Then use --remoteRepositories=https://repo.spring.io/libs-milestone,http://<YourHostIP>:8000 when launching the admin server.

Download and run the Spring Cloud Data Flow shell.

$ wget http://repo.spring.io/snapshot/org/springframework/cloud/spring-cloud-dataflow-shell/1.0.0.BUILD-SNAPSHOT/spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar

$ java -jar spring-cloud-dataflow-shell-1.0.0.BUILD-SNAPSHOT.jar

Deploy a simple stream in the shell

dataflow:>stream create --name ticktock --definition "time | log" --deploy

In the Mesos UI you can then look at the logs for the log sink.

2015-12-29 06:27:29.702  INFO 1 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 8080 (http)
2015-12-29 06:27:29.776  INFO 1 --- [           main] o.s.c.s.module.log.LogSinkApplication    : Started LogSinkApplication in 169.9 seconds (JVM running for 239.813)
2015-12-29 06:27:31.017  INFO 1 --- [hannel-adapter1] log.sink         : 2015-12-29 06:27:28
2015-12-29 06:27:31.506  INFO 1 --- [hannel-adapter1] log.sink         : 2015-12-29 06:27:31
2015-12-29 06:27:32.509  INFO 1 --- [hannel-adapter1] log.sink         : 2015-12-29 06:27:32
2015-12-29 06:27:33.517  INFO 1 --- [hannel-adapter1] log.sink         : 2015-12-29 06:27:33
2015-12-29 06:27:34.525  INFO 1 --- [hannel-adapter1] log.sink         : 2015-12-29 06:27:34