org.springframework.data.hadoop.fs

Class DistCp

java.lang.Object

org.springframework.data.hadoop.fs.DistCp

public class DistCp
extends java.lang.Object

Exposes the Hadoop command-line DistCp functionality as an embeddable API. Due to the number of options available in DistCp, one can either specify them in a command-like style (through one or multiple Strings) through copy(String...) or specify individual arguments through the rest of the methods.

Author:

Costin Leau, Thomas Risberg, Janne Valkealahti

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DistCp.Preserve Enumeration for the possible attributes that can be preserved by a copy operation.

Constructor Summary

Constructors

Constructor and Description
DistCp(org.apache.hadoop.conf.Configuration configuration) Instantiates a new DistCp.
DistCp(org.apache.hadoop.conf.Configuration configuration, java.lang.String user) Instantiates a new DistCp.

Method Summary

Methods

Modifier and Type	Method and Description
void	copy(java.lang.Boolean preserveReplication, java.lang.Boolean preserveBlockSize, java.lang.Boolean preserveUser, java.lang.Boolean preserveGroup, java.lang.Boolean preservePermission, java.lang.Boolean ignoreFailures, java.lang.Boolean skipCrc, java.lang.String logDir, java.lang.Integer mappers, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.Long fileLimit, java.lang.Long sizeLimit, java.lang.String fileList, java.lang.String... uris) Copies the given resources using the given parameters.
void	copy(java.util.EnumSet<DistCp.Preserve> preserve, java.lang.Boolean ignoreFailures, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.String... uris) Copies the given resources using the given parameters.
void	copy(java.util.EnumSet<DistCp.Preserve> preserve, java.lang.Boolean ignoreFailures, java.lang.Boolean skipCrc, java.lang.String logDir, java.lang.Integer mappers, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.Long fileLimit, java.lang.Long sizeLimit, java.lang.String fileList, java.lang.String... uris) Copies the given resources using the given parameters.
void	copy(java.lang.String... arguments) Initiate a copy operation using a command-line style (arguments are specified as Strings).
void	copy(java.lang.String source, java.lang.String destination) Initiate a basic copy operation, between a source and a destination using the defaults.
void	copy(java.lang.String source1, java.lang.String source2, java.lang.String destination) Initiate a basic copy operation, between two sources and a destination using the defaults.
void	setUser(java.lang.String user) Sets the user impersonation (optional) for creating this utility.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistCp

public DistCp(org.apache.hadoop.conf.Configuration configuration)

Instantiates a new DistCp.

Parameters:

configuration - the hadoop configuration

DistCp

public DistCp(org.apache.hadoop.conf.Configuration configuration,
      java.lang.String user)

Instantiates a new DistCp.

Parameters:

configuration - the hadoop configuration

user - the user

Method Detail

copy

public void copy(java.util.EnumSet<DistCp.Preserve> preserve,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.String... uris)

Copies the given resources using the given parameters.

Parameters:

preserve - preserve

ignoreFailures - ignoreFailures

overwrite - overwrite

update - update

delete - delete

uris - uris

copy

public void copy(java.util.EnumSet<DistCp.Preserve> preserve,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean skipCrc,
        java.lang.String logDir,
        java.lang.Integer mappers,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.Long fileLimit,
        java.lang.Long sizeLimit,
        java.lang.String fileList,
        java.lang.String... uris)

Copies the given resources using the given parameters.

Parameters:

preserve - preserve

ignoreFailures - ignoreFailures

skipCrc - skipCrc

logDir - logDir

mappers - mappers

overwrite - overwrite

update - update

delete - delete

fileLimit - fileLimit

sizeLimit - sizeLimit

fileList - fileList

uris - uris

copy

public void copy(java.lang.Boolean preserveReplication,
        java.lang.Boolean preserveBlockSize,
        java.lang.Boolean preserveUser,
        java.lang.Boolean preserveGroup,
        java.lang.Boolean preservePermission,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean skipCrc,
        java.lang.String logDir,
        java.lang.Integer mappers,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.Long fileLimit,
        java.lang.Long sizeLimit,
        java.lang.String fileList,
        java.lang.String... uris)

Copies the given resources using the given parameters.

Parameters:

preserveReplication - preserveReplication

preserveBlockSize - preserveBlockSize

preserveUser - preserveUser

preserveGroup - preserveGroup

preservePermission - preservePermission

ignoreFailures - ignoreFailures

skipCrc - skipCrc

logDir - logDir

mappers - mappers

overwrite - overwrite

update - update

delete - delete

fileLimit - fileLimit

sizeLimit - sizeLimit

fileList - fileList

uris - uris

copy

public void copy(java.lang.String source,
        java.lang.String destination)

Initiate a basic copy operation, between a source and a destination using the defaults.

Parameters:

source - the source

destination - the destination

copy

public void copy(java.lang.String source1,
        java.lang.String source2,
        java.lang.String destination)

Initiate a basic copy operation, between two sources and a destination using the defaults.

Parameters:

source1 - the first source

source2 - the second source

destination - the destination

copy

public void copy(java.lang.String... arguments)

Initiate a copy operation using a command-line style (arguments are specified as Strings).

Parameters:

arguments - the copy arguments

setUser

public void setUser(java.lang.String user)

Sets the user impersonation (optional) for creating this utility. Should be used when running against a Hadoop Kerberos cluster.

Parameters:

user - user/group information