org.springframework.yarn.batch.partition
Class AbstractBatchPartitionHandler

java.lang.Object
  org.springframework.yarn.batch.partition.AbstractBatchPartitionHandler

All Implemented Interfaces:

org.springframework.batch.core.partition.PartitionHandler

Direct Known Subclasses:

HdfsSplitBatchPartitionHandler, StaticBatchPartitionHandler

public abstract class AbstractBatchPartitionHandler
extends java.lang.Object
implements org.springframework.batch.core.partition.PartitionHandler

Base implementation of Spring Batch PartitionHandler handling partitioning for Yarn containers.

Author:

Janne Valkealahti

Constructor Summary

AbstractBatchPartitionHandler()
Instantiates a new batch partition handler.

AbstractBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)
Instantiates a new batch partition handler.

Method Summary

protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint>	createResourceRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions) Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected abstract java.util.Set<org.springframework.batch.core.StepExecution>	createStepExecutionSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter, org.springframework.batch.core.StepExecution stepExecution)
ContainerResolver	getContainerResolver() Gets the container resolver.
java.lang.String	getStepName() Gets the step name.
java.util.Collection<org.springframework.batch.core.StepExecution>	handle(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter, org.springframework.batch.core.StepExecution stepExecution)
void	setBatchAppmaster(AbstractBatchAppmaster batchAppmaster) Sets the batch appmaster.
void	setContainerResolver(ContainerResolver containerResolver) Sets the container resolver.
void	setStepName(java.lang.String stepName) Sets the step name.
protected void	waitCompleteState(org.springframework.batch.core.StepExecution masterStepExecution) Uses CountDownLatch to wait completion status from application master.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractBatchPartitionHandler

public AbstractBatchPartitionHandler()

Instantiates a new batch partition handler.

AbstractBatchPartitionHandler

public AbstractBatchPartitionHandler(AbstractBatchAppmaster batchAppmaster)

Instantiates a new batch partition handler.

Parameters:

batchAppmaster - the batch appmaster

Method Detail

setBatchAppmaster

public void setBatchAppmaster(AbstractBatchAppmaster batchAppmaster)

Sets the batch appmaster.

Parameters:

batchAppmaster - the new batch appmaster

createStepExecutionSplits

protected abstract java.util.Set<org.springframework.batch.core.StepExecution> createStepExecutionSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
                                                                                                         org.springframework.batch.core.StepExecution stepExecution)
                                                                                                  throws java.lang.Exception

Throws:

java.lang.Exception

createResourceRequestData

protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint> createResourceRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions)
                                                                                                              throws java.lang.Exception

Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality. Default implementation returns an empty map.

Parameters:

stepExecutions - Set of step executions

Returns:

Mapping between step executions and container request data

Throws:

java.lang.Exception - If error occurred

handle

public final java.util.Collection<org.springframework.batch.core.StepExecution> handle(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
                                                                                       org.springframework.batch.core.StepExecution stepExecution)
                                                                                throws java.lang.Exception

Specified by:

handle in interface org.springframework.batch.core.partition.PartitionHandler

Throws:

java.lang.Exception

getStepName

public java.lang.String getStepName()

Gets the step name.

Returns:

the step name

setStepName

public void setStepName(java.lang.String stepName)

Sets the step name.

Parameters:

stepName - the new step name

getContainerResolver

public ContainerResolver getContainerResolver()

Gets the container resolver.

Returns:

the container resolver

setContainerResolver

public void setContainerResolver(ContainerResolver containerResolver)

Sets the container resolver.

Parameters:

containerResolver - the new container resolver

waitCompleteState

protected void waitCompleteState(org.springframework.batch.core.StepExecution masterStepExecution)

Uses CountDownLatch to wait completion status from application master. Status is considered to be complete if either master itself or parent step execution sends complete status.

Parameters:

masterStepExecution - the parent step execution