29. Composed Tasks

Spring Cloud Data Flow allows a user to create a directed graph where each node of the graph is a task application. This is done by using the DSL for composed tasks. A composed task can be created via the RESTful API, the Spring Cloud Data Flow Shell, or the Spring Cloud Data Flow UI.

29.1 Configuring the Composed Task Runner

Composed tasks are executed via a task application called the Composed Task Runner.

29.1.1 Registering the Composed Task Runner

Out of the box the Composed Task Runner application is not registered with Spring Cloud Data Flow. So, to launch composed tasks we must first register the Composed Task Runner as an application with Spring Cloud Data Flow as follows:

app register --name composed-task-runner --type task --uri maven://org.springframework.cloud.task.app:composedtaskrunner-task:<DESIRED_VERSION>

You can also configure Spring Cloud Data Flow to use a different task definition name for the composed task runner. This can be done by setting the spring.cloud.dataflow.task.composedTaskRunnerName property to the name of your choice. You can then register the composed task runner application with the name you set using that property.

29.1.2 Configuring the Composed Task Runner

The Composed Task Runner application has a dataflow.server.uri property that is used for validation and for launching child tasks. This defaults to localhost:9393. If you run a distributed Spring Cloud Data Flow server, like you would do if you deploy the server on Cloud Foundry, YARN or Kubernetes, then you need to provide the URI that can be used to access the server. You can either provide this dataflow.server.uri property for the Composed Task Runner application when launching a composed task, or you can provide a spring.cloud.dataflow.server.uri property for the Spring Cloud Data Flow server when it is started. For the latter case the dataflow.server.uri Composed Task Runner application property will be automatically set when a composed task is launched.

29.2 The Lifecycle of a Composed Task

29.2.1 Creating a Composed Task

The DSL for the composed tasks is used when creating a task definition via the task create command. For example:

dataflow:> app register --name timestamp --type task --uri maven://org.springframework.cloud.task.app:timestamp-task:<DESIRED_VERSION>
dataflow:> app register --name mytaskapp --type task --uri file:///home/tasks/mytask.jar
dataflow:> task create my-composed-task --definition "mytaskapp && timestamp"
dataflow:> task launch my-composed-task

In the example above we assume that the applications to be used by our composed task have not been registered yet. So the first two steps we register two task applications. We then create our composed task definition by using the task create command. The composed task DSL in the example above will, when launched, execute mytaskapp and then execute the timestamp application.

But before we launch the my-composed-task definition, we can view what Spring Cloud Data Flow generated for us. This can be done by executing the task list command.

dataflow:>task list
╔══════════════════════════╤══════════════════════╤═══════════╗
║        Task Name         │   Task Definition    │Task Status║
╠══════════════════════════╪══════════════════════╪═══════════╣
║my-composed-task          │mytaskapp && timestamp│unknown    ║
║my-composed-task-mytaskapp│mytaskapp             │unknown    ║
║my-composed-task-timestamp│timestamp             │unknown    ║
╚══════════════════════════╧══════════════════════╧═══════════╝

Spring Cloud Data Flow created three task definitions, one for each of the applications that comprises our composed task (my-composed-task-mytaskapp and my-composed-task-timestamp) as well as the composed task (my-composed-task) definition. We also see that each of the generated names for the child tasks is comprised of the name of the composed task and the name of the application separated by a dash -. i.e. my-composed-task - mytaskapp.

Task Application Parameters

The task applications that comprise the composed task definition can also contain parameters. For example:

dataflow:> task create my-composed-task --definition "mytaskapp --displayMessage=hello && timestamp --format=YYYY"

29.2.2 Launching a Composed Task

Launching a composed task is done the same way as launching a stand-alone task. i.e.

task launch my-composed-task

Once the task is launched and assuming all the tasks complete successfully you will see three task executions when executing a task execution list. For example:

dataflow:>task execution list
╔══════════════════════════╤═══╤════════════════════════════╤════════════════════════════╤═════════╗
║        Task Name         │ID │         Start Time         │          End Time          │Exit Code║
╠══════════════════════════╪═══╪════════════════════════════╪════════════════════════════╪═════════╣
║my-composed-task-timestamp│713│Wed Apr 12 16:43:07 EDT 2017│Wed Apr 12 16:43:07 EDT 20170        ║
║my-composed-task-mytaskapp│712│Wed Apr 12 16:42:57 EDT 2017│Wed Apr 12 16:42:57 EDT 20170        ║
║my-composed-task          │711│Wed Apr 12 16:42:55 EDT 2017│Wed Apr 12 16:43:15 EDT 20170        ║
╚══════════════════════════╧═══╧════════════════════════════╧════════════════════════════╧═════════╝

In the example above we see that my-compose-task launched and it also launched the other tasks in sequential order and all of them executed successfully with "Exit Code" as 0.

Exit Statuses

The following list shows how the Exit Status will be set for each step (task) contained in the composed task following each step execution.

  • If the TaskExecution has an ExitMessage that will be used as the ExitStatus
  • If no ExitMessage is present and the ExitCode is set to zero then the ExitStatus for the step will be COMPLETED.
  • If no ExitMessage is present and the ExitCode is set to any non zero number then the ExitStatus for the step will be FAILED.

29.2.3 Destroying a Composed Task

The same command used to destroy a stand-alone task is the same as destroying a composed task. The only difference is that destroying a composed task will also destroy the child tasks associated with it. For example

dataflow:>task list
╔══════════════════════════╤══════════════════════╤═══════════╗
║        Task Name         │   Task Definition    │Task Status║
╠══════════════════════════╪══════════════════════╪═══════════╣
║my-composed-task          │mytaskapp && timestamp│COMPLETED  ║
║my-composed-task-mytaskapp│mytaskapp             │COMPLETED  ║
║my-composed-task-timestamp│timestamp             │COMPLETED  ║
╚══════════════════════════╧══════════════════════╧═══════════╝
...
dataflow:>task destroy my-composed-task
dataflow:>task list
╔═════════╤═══════════════╤═══════════╗
║Task Name│Task Definition│Task Status║
╚═════════╧═══════════════╧═══════════╝

29.2.4 Stopping a Composed Task

In cases where a composed task execution needs to be stopped. This can be done via the:

  • RESTful API
  • Spring Cloud Data Flow Dashboard by selecting the Job’s tab and then clicking the stop button by the job execution that needs to be stopped.

The composed task run will be stopped when the currently running child task completes. The step associated with the child task that was running at the time that the composed task was stopped will be marked as STOPPED as well as the composed task job execution.

29.2.5 Restarting a Composed Task

In cases where a composed task fails during execution and the status of the composed task is FAILED then the task can be restarted. This can be done via the:

  • RESTful API
  • Shell by launching the task using the same parameters
  • Spring Cloud Data Flow Dashboard by selecting the Job’s tab and then clicking the restart button by the job execution that needs to be restarted.
[Note]Note

Restarting a Composed Task job that has been stopped (via the Spring Cloud Data Flow Dashboard or RESTful API), will relaunch the STOPPED child task, and then launch the remaining (unlaunched) child tasks in the specified order.