10. Remote Partitioning

Spring Cloud Deployer provides facilities for launching Spring Boot based applications on most cloud infrastructures. The DeployerPartitionHandler and DeployerStepExecutionHandler delegate the launching of worker step executions to Spring Cloud Deployer.

To configure the DeployerStepExecutionHandler, a Resource representing the Spring Boot über-jar to be executed, a TaskLauncher, and a JobExplorer are all required. You can configure any environment properties as well as the max number of workers to be executing at once, the interval to poll for the results (defaults to 10 seconds), and a timeout (defaults to -1 or no timeout). An example of configuring this PartitionHandler would look like the following:

@Bean
public PartitionHandler partitionHandler(TaskLauncher taskLauncher,
		JobExplorer jobExplorer) throws Exception {

	MavenProperties mavenProperties = new MavenProperties();
	mavenProperties.setRemoteRepositories(new HashMap<>(Collections.singletonMap("springRepo",
		new MavenProperties.RemoteRepository(repository))));

 	Resource resource =
		MavenResource.parse(String.format("%s:%s:%s",
				"io.spring.cloud",
				"partitioned-batch-job",
				"1.1.0.M2"), mavenProperties);

	DeployerPartitionHandler partitionHandler =
		new DeployerPartitionHandler(taskLauncher, jobExplorer, resource, "workerStep");

	List<String> commandLineArgs = new ArrayList<>(3);
	commandLineArgs.add("--spring.profiles.active=worker");
	commandLineArgs.add("--spring.cloud.task.initialize.enable=false");
	commandLineArgs.add("--spring.batch.initializer.enabled=false");

	partitionHandler.setCommandLineArgsProvider(new PassThroughCommandLineArgsProvider(commandLineArgs));
	partitionHandler.setEnvironmentVariablesProvider(new NoOpEnvironmentVariablesProvider());
	partitionHandler.setMaxWorkers(2);
	partitionHandler.setApplicationName("PartitionedBatchJobTask");

	return partitionHandler;
}
[Note]Note

When passing environment variables to partitions, each partition may be on a different machine with a different environment settings. So only pass those that are required.

The Resource to be executed is expected to be a Spring Boot über-jar with a DeployerStepExecutionHandler configured as a CommandLineRunner in the current context. The repository enumerated in the example above should be the location of the remote repository from which the über-jar is located. Both the master and slave are expected to have visibility into the same data store being used as the job repository and task repository. Once the underlying infrastructure has bootstrapped the Spring Boot jar and Spring Boot has launched the DeployerStepExecutionHandler, the step handler will execute the Step requested. An example of configuring the DefaultStepExecutionHandler is show below:

@Bean
public DeployerStepExecutionHandler stepExecutionHandler(JobExplorer jobExplorer) {
	DeployerStepExecutionHandler handler =
		new DeployerStepExecutionHandler(this.context, jobExplorer, this.jobRepository);

	return handler;
}
[Note]Note

A sample remote partition application can be found in the samples module of the Spring Cloud Task Project here.

10.1 Notes on developing a batch partitioned app for the Yarn platform

  • When deploying partitioned apps on the Yarn platform be sure use the following dependency for the Spring Cloud Yarn Deployer (with a version 1.0.2 or higher):
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-deployer-yarn</artifactId>
		</dependency>
  • Add the following dependency to the dependency management for a transient dependency required by Yarn:
	<dependencyManagement>
		<dependencies>
...
			<dependency>
				<groupId>com.google.guava</groupId>
				<artifactId>guava</artifactId>
				<version>18.0</version>
			</dependency>
		</dependencies>
...
	</dependencyManagement>
  • Also add the following property to your application.properties: spring.yarn.container.keepContextAlive=false.
  • When setting up environment variables for the partitions in the PartitionHandler it is recommended that you do not copy the current working environment properties.

10.2 Notes on developing a batch partitioned app for the Kubernetes platform

Application name for the task application and its partitions need to follow the following regex pattern [a-z0-9]([-a-z0-9]*[a-z0-9]). Else an exception will be thrown.