org.springframework.yarn.batch.partition

Class AbstractPartitionHandler

java.lang.Object

org.springframework.yarn.batch.partition.AbstractPartitionHandler

All Implemented Interfaces:

org.springframework.batch.core.partition.PartitionHandler

Direct Known Subclasses:

SplitterPartitionHandler, StaticPartitionHandler

public abstract class AbstractPartitionHandler
extends java.lang.Object
implements org.springframework.batch.core.partition.PartitionHandler

Base implementation of Spring Batch PartitionHandler handling partitioning for Yarn containers.

Author:

Janne Valkealahti

Constructor Summary

Constructors

Constructor and Description
AbstractPartitionHandler() Instantiates a new abstract partition handler.
AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster) Instantiates a new abstract partition handler.

Method Summary

Methods

Modifier and Type	Method and Description
protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint>	createRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions) Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution.
protected abstract java.util.Set<org.springframework.batch.core.StepExecution>	createSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter, org.springframework.batch.core.StepExecution stepExecution) Creates the splits.
java.lang.String	getKeySplitLocations() Gets the key split locations.
java.lang.String	getStepName() Gets the remote step name.
java.util.Collection<org.springframework.batch.core.StepExecution>	handle(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter, org.springframework.batch.core.StepExecution stepExecution)
void	setBatchAppmaster(BatchYarnAppmaster batchAppmaster) Sets the batch appmaster.
void	setKeySplitLocations(java.lang.String keySplitLocations) The name of the key for the split locations in each ExecutionContext.
void	setStepName(java.lang.String stepName) Sets the remote step name.
void	setYarnAppmaster(YarnAppmaster yarnAppmaster) Sets the batch appmaster.
protected void	waitCompleteState(org.springframework.batch.core.StepExecution masterStepExecution) Uses CountDownLatch to wait completion status from application master.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractPartitionHandler

public AbstractPartitionHandler()

Instantiates a new abstract partition handler.

AbstractPartitionHandler

public AbstractPartitionHandler(BatchYarnAppmaster batchAppmaster)

Instantiates a new abstract partition handler.

Parameters:

batchAppmaster - the batch appmaster

Method Detail

handle

public final java.util.Collection<org.springframework.batch.core.StepExecution> handle(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
                                                                        org.springframework.batch.core.StepExecution stepExecution)
                                                                                throws java.lang.Exception

Specified by:

handle in interface org.springframework.batch.core.partition.PartitionHandler

Throws:

java.lang.Exception

setBatchAppmaster

public void setBatchAppmaster(BatchYarnAppmaster batchAppmaster)

Sets the batch appmaster.

Parameters:

batchAppmaster - the new batch appmaster

See Also:

setYarnAppmaster(YarnAppmaster)

setYarnAppmaster

public void setYarnAppmaster(YarnAppmaster yarnAppmaster)

Sets the batch appmaster. This is a specific method using a YarnAppmaster to be able to allow application context to auto-wire BatchYarnAppmaster if it's type in context is defined as YarnAppmaster.

Parameters:

yarnAppmaster - the new yarn appmaster

See Also:

setBatchAppmaster(BatchYarnAppmaster)

setKeySplitLocations

public void setKeySplitLocations(java.lang.String keySplitLocations)

The name of the key for the split locations in each ExecutionContext. Defaults to "splitLocations".

Parameters:

keySplitLocations - the value of the key

getKeySplitLocations

public java.lang.String getKeySplitLocations()

Gets the key split locations.

Returns:

the key split locations

getStepName

public java.lang.String getStepName()

Gets the remote step name.

Returns:

the remote step name

setStepName

public void setStepName(java.lang.String stepName)

Sets the remote step name.

Parameters:

stepName - the new remote step name

createSplits

protected abstract java.util.Set<org.springframework.batch.core.StepExecution> createSplits(org.springframework.batch.core.partition.StepExecutionSplitter stepSplitter,
                                                                       org.springframework.batch.core.StepExecution stepExecution)
                                                                                     throws java.lang.Exception

Creates the splits. Implementor needs to override this method to create an actual step splits.

Parameters:

stepSplitter - the step splitter

stepExecution - the step execution

Returns:

the step executions

Throws:

java.lang.Exception - the exception

createRequestData

protected java.util.Map<org.springframework.batch.core.StepExecution,ContainerRequestHint> createRequestData(java.util.Set<org.springframework.batch.core.StepExecution> stepExecutions)
                                                                                                      throws java.lang.Exception

Subclass may override this method to assign a specific ContainerRequestHint to a StepExecution. This would be needed in cases where step should be executed in a specific host or rack considering data locality.

Default implementation returns an empty map.

Parameters:

stepExecutions - Set of step executions

Returns:

Mapping between step executions and container request data

Throws:

java.lang.Exception - If error occurred

waitCompleteState

protected void waitCompleteState(org.springframework.batch.core.StepExecution masterStepExecution)

Uses CountDownLatch to wait completion status from application master. Status is considered to be complete if either master itself or parent step execution sends complete status.

Parameters:

masterStepExecution - the parent step execution