org.springframework.yarn.fs

Class AbstractResourceLocalizer

java.lang.Object

org.springframework.yarn.fs.AbstractResourceLocalizer

All Implemented Interfaces:

ResourceLocalizer, SmartResourceLocalizer

Direct Known Subclasses:

DefaultResourceLocalizer

public abstract class AbstractResourceLocalizer
extends java.lang.Object
implements SmartResourceLocalizer

Base implementation of ResourceLocalizer.

Author:

Janne Valkealahti

Constructor Summary

Constructors

Constructor and Description
AbstractResourceLocalizer(org.apache.hadoop.conf.Configuration configuration) Instantiates a new abstract resource localizer.
AbstractResourceLocalizer(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path stagingDirectory) Instantiates a new abstract resource localizer.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	clean() Cleans all leftovers what has been created during the distribute process.
void	copy() If resource localizer has knowledge to copy files into file system, calling this method effectively should do all tasks for that specific functionality.
protected boolean	deleteStagingEntries() Delete staging entries.
void	distribute() If underlying implementation needs to do operations on file system or any other preparation work, calling of this method should make implementation ready to return resources from ResourceLocalizer.getResources() command.
protected abstract void	doFileCopy(org.apache.hadoop.fs.FileSystem fs) Do file copy.
protected abstract java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	doFileTransfer(org.apache.hadoop.fs.FileSystem fs) Do file transfer.
org.apache.hadoop.conf.Configuration	getConfiguration() Gets the hadoop configuration.
java.util.concurrent.locks.ReentrantLock	getLock() Gets the lock.
java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	getResources() Gets a map of LocalResource instances.
protected boolean	isCopied() Checks if is copied.
protected boolean	isDistributed() Checks if is distributed.
void	resolve() Manually resolve resources before distribution process is initiated.
protected org.apache.hadoop.fs.Path	resolveStagingDirectory() Resolve runtime staging directory.
protected void	setCopied(boolean copied) Sets the copied.
protected void	setDistributed(boolean distributed) Sets the distributed.
void	setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory) Sets the staging directory.
void	setStagingId(java.lang.String stagingId) Sets the staging id.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractResourceLocalizer

public AbstractResourceLocalizer(org.apache.hadoop.conf.Configuration configuration)

Instantiates a new abstract resource localizer.

Parameters:

configuration - the configuration

AbstractResourceLocalizer

public AbstractResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.Path stagingDirectory)

Instantiates a new abstract resource localizer.

Parameters:

configuration - the configuration

stagingDirectory - the staging directory

Method Detail

getResources

public java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> getResources()

Description copied from interface: ResourceLocalizer

Gets a map of LocalResource instances. Underlying instances of LocalResources needs to be fully initialised including resource size and timestamp.

Calling this method should do all needed preparation work needed to return LocalResources.

Specified by:

getResources in interface ResourceLocalizer

Returns:

The map containing LocalResource instances

copy

public void copy()

Description copied from interface: SmartResourceLocalizer

If resource localizer has knowledge to copy files into file system, calling this method effectively should do all tasks for that specific functionality.

Specified by:

copy in interface SmartResourceLocalizer

distribute

public void distribute()

Description copied from interface: SmartResourceLocalizer

If underlying implementation needs to do operations on file system or any other preparation work, calling of this method should make implementation ready to return resources from ResourceLocalizer.getResources() command.

Effectively result of calling this method should be same as calling both SmartResourceLocalizer.copy() and SmartResourceLocalizer.resolve() methods manually.

Specified by:

distribute in interface SmartResourceLocalizer

resolve

public void resolve()

Description copied from interface: SmartResourceLocalizer

Manually resolve resources before distribution process is initiated. Using this method should always be optional and executed before distribution if it has not already been done.

Specified by:

resolve in interface SmartResourceLocalizer

clean

public boolean clean()

Description copied from interface: SmartResourceLocalizer

Cleans all leftovers what has been created during the distribute process. These can i.e. be files in staging directory.

Specified by:

clean in interface SmartResourceLocalizer

Returns:

true, if cleanup were attempted and was successful

setStagingDirectory

public void setStagingDirectory(org.apache.hadoop.fs.Path stagingDirectory)

Description copied from interface: SmartResourceLocalizer

Sets the staging directory. If not set, path is determined by the implementation. Default value can i.e. be user's hadoop home directory.

Specified by:

setStagingDirectory in interface SmartResourceLocalizer

Parameters:

stagingDirectory - the new staging directory

setStagingId

public void setStagingId(java.lang.String stagingId)

Description copied from interface: SmartResourceLocalizer

Sets the staging id. Id is used together with Path set in SmartResourceLocalizer.setStagingDirectory(Path) to post fix unique runtime staging path. If not set simultaneous instances of same application may override files.

Specified by:

setStagingId in interface SmartResourceLocalizer

Parameters:

stagingId - the new staging id

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

Gets the hadoop configuration.

Returns:

the hadoop configuration

getLock

public java.util.concurrent.locks.ReentrantLock getLock()

Gets the lock.

Returns:

the lock

doFileCopy

protected abstract void doFileCopy(org.apache.hadoop.fs.FileSystem fs)
                            throws java.lang.Exception

Do file copy.

Parameters:

fs - the fs

Throws:

java.lang.Exception - the exception

doFileTransfer

protected abstract java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> doFileTransfer(org.apache.hadoop.fs.FileSystem fs)
                                                                                                            throws java.lang.Exception

Do file transfer.

Parameters:

fs - the fs

Returns:

the map

Throws:

java.lang.Exception - the exception

isDistributed

protected boolean isDistributed()

Checks if is distributed.

Returns:

true, if is distributed

setDistributed

protected void setDistributed(boolean distributed)

Sets the distributed.

Parameters:

distributed - the new distributed

isCopied

protected boolean isCopied()

Checks if is copied.

Returns:

true, if is copied

setCopied

protected void setCopied(boolean copied)

Sets the copied.

Parameters:

copied - the new copied

resolveStagingDirectory

protected org.apache.hadoop.fs.Path resolveStagingDirectory()

Resolve runtime staging directory.

Returns:

the resolved path of runtime stagind directory

deleteStagingEntries

protected boolean deleteStagingEntries()

Delete staging entries.

Returns:

true, if successful