org.springframework.yarn.fs
Class DefaultResourceLocalizer

java.lang.Object
  org.springframework.yarn.fs.DefaultResourceLocalizer

All Implemented Interfaces:

ResourceLocalizer

public class DefaultResourceLocalizer
extends java.lang.Object
implements ResourceLocalizer

Default implementation of ResourceLocalizer which is only capable of re-using files already in HDFS and preparing correct parameters for created LocalResource entries.

Author:

Janne Valkealahti

Constructor Summary

DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration, java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries, java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries)
Instantiates a new default resource localizer.

DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration, java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries, java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries, org.apache.hadoop.fs.Path stagingDirectory)
Instantiates a new default resource localizer.

Method Summary

boolean	clean() Cleans all leftovers what has been created during the distribute process.
void	distribute() If underlying implementation needs to do operations on hdfs filesystem or any other preparation work, calling of this method should make implementation ready to return resources from ResourceLocalizer.getResources() command.
protected void	doFileCopy(org.apache.hadoop.fs.FileSystem fs) Do file copy.
protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	doFileTransfer(org.apache.hadoop.fs.FileSystem fs) Gets a map of localized resources.
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	getResources() Gets a map of LocalResource instances.
void	setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory) Sets the staging directory.
void	setStagingId(java.lang.String stagingId) Sets the staging id.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultResourceLocalizer

public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
                                java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
                                java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries)

Instantiates a new default resource localizer.

Parameters:

configuration - the configuration

transferEntries - the transfer entries

copyEntries - the copy entries

DefaultResourceLocalizer

public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
                                java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
                                java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries,
                                org.apache.hadoop.fs.Path stagingDirectory)

Instantiates a new default resource localizer.

Parameters:

configuration - the configuration

transferEntries - the transfer entries

copyEntries - the copy entries

stagingDirectory - the staging directory

Method Detail

getResources

public java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> getResources()

Description copied from interface: ResourceLocalizer

Gets a map of LocalResource instances. Underlying instances of LocalResources needs to be fully initialised including resource size and timestamp.

Specified by:

getResources in interface ResourceLocalizer

Returns:

The map containing LocalResource instances

setStagingDirectory

public void setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory)

Description copied from interface: ResourceLocalizer

Sets the staging directory. If not set, path is determined by the implementation. Default value can i.e. be user's hadoop home directory.

Specified by:

setStagingDirectory in interface ResourceLocalizer

Parameters:

stagingDirectory - the new staging directory

setStagingId

public void setStagingId(java.lang.String stagingId)

Description copied from interface: ResourceLocalizer

Sets the staging id. Id is used together with Path set in ResourceLocalizer.setStagingDirectory(Path) to post fix unique runtime staging path. If not set simultaneous instances of same application may override files.

Specified by:

setStagingId in interface ResourceLocalizer

Parameters:

stagingId - the new staging id

distribute

public void distribute()

Description copied from interface: ResourceLocalizer

If underlying implementation needs to do operations on hdfs filesystem or any other preparation work, calling of this method should make implementation ready to return resources from ResourceLocalizer.getResources() command.

Specified by:

distribute in interface ResourceLocalizer

clean

public boolean clean()

Description copied from interface: ResourceLocalizer

Cleans all leftovers what has been created during the distribute process. These can i.e. be files in staging directory.

Specified by:

clean in interface ResourceLocalizer

Returns:

true, if successful

doFileCopy

protected void doFileCopy(org.apache.hadoop.fs.FileSystem fs)
                   throws java.io.IOException

Do file copy.

Parameters:

fs - the fs

Throws:

java.io.IOException - Signals that an I/O exception has occurred.

doFileTransfer

protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> doFileTransfer(org.apache.hadoop.fs.FileSystem fs)
                                                                                                   throws java.io.IOException,
                                                                                                          java.net.URISyntaxException

Gets a map of localized resources.

Parameters:

fs - the file system

Returns:

a map of localized resources

Throws:

java.io.IOException - if problem occurred getting file status

java.net.URISyntaxException - if file path is wrong