|
Spring for Apache Hadoop | ||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.springframework.yarn.batch.partition.MultiHdfsResourcePartitioner
public class MultiHdfsResourcePartitioner
Implementation of Partitioner
that locates multiple resources and
associates their file names with execution context keys. Creates an
ExecutionContext
per resource, and labels them as
{partition0, partition1, ..., partitionN}
. The grid size is
ignored.
Constructor Summary | |
---|---|
MultiHdfsResourcePartitioner()
|
Method Summary | |
---|---|
java.util.Map<java.lang.String,org.springframework.batch.item.ExecutionContext> |
partition(int gridSize)
Assign the filename of each of the injected resources to an ExecutionContext . |
void |
setConfiguration(org.apache.hadoop.conf.Configuration configuration)
Sets the Yarn configuration. |
void |
setForceSplit(boolean forceSplit)
Sets the force split. |
void |
setKeyFileName(java.lang.String keyFileName)
The name of the key for the file name in each ExecutionContext . |
void |
setKeySplitLength(java.lang.String keySplitLength)
The name of the key for the file split length in each ExecutionContext . |
void |
setKeySplitStart(java.lang.String keySplitStart)
The name of the key for the file split start in each ExecutionContext . |
void |
setResources(org.springframework.core.io.Resource[] resources)
The resources to assign to each partition. |
void |
setSplitFile(boolean splitFile)
Sets the flat indicating if file input should be split. |
void |
setSplitSize(int splitSize)
Sets the input split size relative to block size of the HDFS file. |
void |
setUsePath(boolean usePath)
If set to true resource path is set using URL.getPath() , otherwise
URL.toExternalForm() is used. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public MultiHdfsResourcePartitioner()
Method Detail |
---|
public void setResources(org.springframework.core.io.Resource[] resources)
resources
- the resources to usepublic void setKeyFileName(java.lang.String keyFileName)
ExecutionContext
.
Defaults to "fileName".
keyFileName
- the value of the keypublic void setKeySplitStart(java.lang.String keySplitStart)
ExecutionContext
.
Defaults to "splitStart".
keySplitStart
- the value of the keypublic void setKeySplitLength(java.lang.String keySplitLength)
ExecutionContext
.
Defaults to "splitLength".
keySplitLength
- the value of the keypublic void setSplitSize(int splitSize)
splitSize
- the new split sizepublic void setSplitFile(boolean splitFile)
setSplitSize(int)
.
Default value is TRUE
splitFile
- the new split filepublic void setForceSplit(boolean forceSplit)
FALSE
input is forced to split if file size is
below HDFS block size. Useful for testing and
cases where processed data is very cpu intensive.
Default value is FALSE
forceSplit
- the new force splitpublic void setUsePath(boolean usePath)
URL.getPath()
, otherwise
URL.toExternalForm()
is used.
Default value is TRUE
.
usePath
- whether path part is usedpublic void setConfiguration(org.apache.hadoop.conf.Configuration configuration)
configuration
- the new Yarn configurationpublic java.util.Map<java.lang.String,org.springframework.batch.item.ExecutionContext> partition(int gridSize)
ExecutionContext
.
partition
in interface org.springframework.batch.core.partition.support.Partitioner
Partitioner.partition(int)
|
Spring for Apache Hadoop | ||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |