org.springframework.yarn.fs

Class DefaultResourceLocalizer

java.lang.Object

org.springframework.yarn.fs.AbstractResourceLocalizer

org.springframework.yarn.fs.DefaultResourceLocalizer

All Implemented Interfaces:

ResourceLocalizer, SmartResourceLocalizer

public class DefaultResourceLocalizer
extends AbstractResourceLocalizer
implements SmartResourceLocalizer

Default implementation of ResourceLocalizer which is only capable of re-using files already in HDFS and preparing correct parameters for created LocalResource entries.

Author:

Janne Valkealahti

Constructor Summary

Constructors

Constructor and Description
DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration, java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries, java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries) Instantiates a new default resource localizer.
DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration, java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries, java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries, org.apache.hadoop.fs.Path stagingDirectory) Instantiates a new default resource localizer.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	AddRawContent(java.lang.String key, byte[] value) Adds a content into a to be written entries.
protected void	doFileCopy(org.apache.hadoop.fs.FileSystem fs) Do file copy.
protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	doFileTransfer(org.apache.hadoop.fs.FileSystem fs) Do file transfer.
void	setRawFileContents(java.util.Map<java.lang.String,byte[]> rawFileContents) Sets the raw file contents.

Methods inherited from class org.springframework.yarn.fs.AbstractResourceLocalizer

clean, copy, deleteStagingEntries, distribute, getConfiguration, getLock, getResources, isCopied, isDistributed, resolve, resolveStagingDirectory, setCopied, setDistributed, setStagingDirectory, setStagingId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.yarn.fs.SmartResourceLocalizer

clean, copy, distribute, resolve, setStagingDirectory, setStagingId

Methods inherited from interface org.springframework.yarn.fs.ResourceLocalizer

getResources

Constructor Detail

DefaultResourceLocalizer

public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
                        java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
                        java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries)

Instantiates a new default resource localizer.

Parameters:

configuration - the configuration

transferEntries - the transfer entries

copyEntries - the copy entries

DefaultResourceLocalizer

public DefaultResourceLocalizer(org.apache.hadoop.conf.Configuration configuration,
                        java.util.Collection<LocalResourcesFactoryBean.TransferEntry> transferEntries,
                        java.util.Collection<LocalResourcesFactoryBean.CopyEntry> copyEntries,
                        org.apache.hadoop.fs.Path stagingDirectory)

Instantiates a new default resource localizer.

Parameters:

configuration - the configuration

transferEntries - the transfer entries

copyEntries - the copy entries

stagingDirectory - the staging directory

Method Detail

AddRawContent

public boolean AddRawContent(java.lang.String key,
                    byte[] value)

Adds a content into a to be written entries.

Parameters:

key - the key considered as a file name

value - the content of a file to be written

Returns:

true, existing content for key already exist and was replaced

setRawFileContents

public void setRawFileContents(java.util.Map<java.lang.String,byte[]> rawFileContents)

Sets the raw file contents. Overwrites all existing contents.

Parameters:

rawFileContents - the raw file contents

doFileCopy

protected void doFileCopy(org.apache.hadoop.fs.FileSystem fs)
                   throws java.lang.Exception

Description copied from class: AbstractResourceLocalizer

Do file copy.

Specified by:

doFileCopy in class AbstractResourceLocalizer

Parameters:

fs - the fs

Throws:

java.lang.Exception - the exception

doFileTransfer

protected java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> doFileTransfer(org.apache.hadoop.fs.FileSystem fs)
                                                                                                   throws java.lang.Exception

Description copied from class: AbstractResourceLocalizer

Do file transfer.

Specified by:

doFileTransfer in class AbstractResourceLocalizer

Parameters:

fs - the fs

Returns:

the map

Throws:

java.lang.Exception - the exception