org.springframework.yarn.batch.partition

Class SplitterPartitionHandler

java.lang.Object

org.springframework.yarn.batch.partition.AbstractPartitionHandler

org.springframework.yarn.batch.partition.SplitterPartitionHandler

All Implemented Interfaces:

org.springframework.batch.core.partition.PartitionHandler

public class SplitterPartitionHandler
extends AbstractPartitionHandler

Implementation of Spring Batch PartitionHandler which does partitioning based on number of input files from HDFS.

Author:

Janne Valkealahti

Constructor Summary

Constructors

Constructor and Description
SplitterPartitionHandler() Instantiates a new splitter partition handler.
SplitterPartitionHandler(AbstractBatchAppmaster batchAppmaster) Instantiates a new splitter partition handler.

Method Summary

Methods

Modifier and Type	Method and Description
protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint>	createRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions) Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected java.util.Set<org.springframework.batch.core.StepExecution>	createSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter, org.springframework.batch.core.StepExecution stepExecution) Creates the splits.

Methods inherited from class org.springframework.yarn.batch.partition.AbstractPartitionHandler

getKeySplitLocations, getStepName, handle, setBatchAppmaster, setKeySplitLocations, setStepName, setYarnAppmaster, waitCompleteState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SplitterPartitionHandler

public SplitterPartitionHandler()

Instantiates a new splitter partition handler.

SplitterPartitionHandler

public SplitterPartitionHandler(AbstractBatchAppmaster batchAppmaster)

Instantiates a new splitter partition handler.

Parameters:

batchAppmaster - the batch appmaster

Method Detail

createSplits

protected java.util.Set<org.springframework.batch.core.StepExecution> createSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
                                                                       org.springframework.batch.core.StepExecution stepExecution)
                                                                            throws java.lang.Exception

Description copied from class: AbstractPartitionHandler

Creates the splits. Implementor needs to override this method to create an actual step splits.

Specified by:

createSplits in class AbstractPartitionHandler

Parameters:

stepSplitter - the step splitter

stepExecution - the step execution

Returns:

the step executions

Throws:

java.lang.Exception - the exception

createRequestData

protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint> createRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions)
                                                                                                      throws java.lang.Exception

Description copied from class: AbstractPartitionHandler

Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality.

Default implementation returns an empty map.

Overrides:

createRequestData in class AbstractPartitionHandler

Parameters:

stepExecutions - Set of step executions

Returns:

Mapping between step executions and container request data

Throws:

java.lang.Exception - If error occurred