org.springframework.data.hadoop.fs

Class DistCp

java.lang.Object

org.springframework.data.hadoop.fs.DistCp

public class DistCp
extends java.lang.Object

Exposes the Hadoop command-line distcp as an embeddable API. Due to the number of options available in DistCp, one can either specify them in a command-like style (through one or multiple Strings) through copy(String...) or specify individual arguments through the rest of the methods.

Author:

Costin Leau, Thomas Risberg

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DistCp.Preserve Enumeration for the possible attributes that can be preserved by a copy operation.

Constructor Summary

Constructors

Constructor and Description
DistCp(org.apache.hadoop.conf.Configuration configuration) Constructs a new DistCp instance.
DistCp(org.apache.hadoop.conf.Configuration configuration, java.lang.String user)

Method Summary

Methods

Modifier and Type	Method and Description
void	copy(java.lang.Boolean preserveReplication, java.lang.Boolean preserveBlockSize, java.lang.Boolean preserveUser, java.lang.Boolean preserveGroup, java.lang.Boolean preservePermission, java.lang.Boolean ignoreFailures, java.lang.Boolean skipCrc, java.lang.String logDir, java.lang.Integer mappers, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.Long fileLimit, java.lang.Long sizeLimit, java.lang.String fileList, java.lang.String... uris) Copies the given resources using the given parameters.
void	copy(java.util.EnumSet<DistCp.Preserve> preserve, java.lang.Boolean ignoreFailures, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.String... uris)
void	copy(java.util.EnumSet<DistCp.Preserve> preserve, java.lang.Boolean ignoreFailures, java.lang.Boolean skipCrc, java.lang.String logDir, java.lang.Integer mappers, java.lang.Boolean overwrite, java.lang.Boolean update, java.lang.Boolean delete, java.lang.Long fileLimit, java.lang.Long sizeLimit, java.lang.String fileList, java.lang.String... uris) Copies the given resources using the given parameters.
void	copy(java.lang.String... arguments) DistCopy using a command-line style (arguments are specified as Strings).
void	copy(java.lang.String arg1, java.lang.String arg2) Basic copy operation, between a source and a destination using the defaults.
void	copy(java.lang.String arg1, java.lang.String arg2, java.lang.String arg3) Basic copy operation, between a source and a destination using the defaults.
void	setUser(java.lang.String user) Sets the user impersonation (optional) for creating this utility.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistCp

public DistCp(org.apache.hadoop.conf.Configuration configuration)

Constructs a new DistCp instance.

Parameters:

configuration - Hadoop configuration to use.

DistCp

public DistCp(org.apache.hadoop.conf.Configuration configuration,
      java.lang.String user)

Method Detail

copy

public void copy(java.util.EnumSet<DistCp.Preserve> preserve,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.String... uris)

Parameters:

preserve - preserve

ignoreFailures - ignoreFailures

overwrite - overwrite

update - update

delete - delete

uris - uris

copy

public void copy(java.util.EnumSet<DistCp.Preserve> preserve,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean skipCrc,
        java.lang.String logDir,
        java.lang.Integer mappers,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.Long fileLimit,
        java.lang.Long sizeLimit,
        java.lang.String fileList,
        java.lang.String... uris)

Copies the given resources using the given parameters.

Parameters:

preserve - preserve

ignoreFailures - ignoreFailures

skipCrc - skipCrc

logDir - logDir

mappers - mappers

overwrite - overwrite

update - update

delete - delete

fileLimit - fileLimit

sizeLimit - sizeLimit

fileList - fileList

uris - uris

copy

public void copy(java.lang.Boolean preserveReplication,
        java.lang.Boolean preserveBlockSize,
        java.lang.Boolean preserveUser,
        java.lang.Boolean preserveGroup,
        java.lang.Boolean preservePermission,
        java.lang.Boolean ignoreFailures,
        java.lang.Boolean skipCrc,
        java.lang.String logDir,
        java.lang.Integer mappers,
        java.lang.Boolean overwrite,
        java.lang.Boolean update,
        java.lang.Boolean delete,
        java.lang.Long fileLimit,
        java.lang.Long sizeLimit,
        java.lang.String fileList,
        java.lang.String... uris)

Copies the given resources using the given parameters.

Parameters:

preserveReplication - preserveReplication

preserveBlockSize - preserveBlockSize

preserveUser - preserveUser

preserveGroup - preserveGroup

preservePermission - preservePermission

ignoreFailures - ignoreFailures

skipCrc - skipCrc

logDir - logDir

mappers - mappers

overwrite - overwrite

update - update

delete - delete

fileLimit - fileLimit

sizeLimit - sizeLimit

fileList - fileList

uris - uris

copy

public void copy(java.lang.String arg1,
        java.lang.String arg2)

Basic copy operation, between a source and a destination using the defaults.

Parameters:

arg1 - arg1

arg2 - arg2

copy

public void copy(java.lang.String arg1,
        java.lang.String arg2,
        java.lang.String arg3)

Basic copy operation, between a source and a destination using the defaults.

Parameters:

arg1 - arg1

arg2 - arg2

arg3 - arg3

copy

public void copy(java.lang.String... arguments)

DistCopy using a command-line style (arguments are specified as Strings).

Parameters:

arguments - copy arguments

setUser

public void setUser(java.lang.String user)

Sets the user impersonation (optional) for creating this utility. Should be used when running against a Hadoop Kerberos cluster.

Parameters:

user - user/group information