org.springframework.data.hadoop.store.output

Class AbstractPartitionDataStoreWriter<T,K>

java.lang.Object

org.springframework.data.hadoop.store.support.LifecycleObjectSupport

org.springframework.data.hadoop.store.output.AbstractPartitionDataStoreWriter<T,K>

Type Parameters:

T - the type of an entity to write

K - the type of a partition key

All Implemented Interfaces:

java.io.Closeable, java.io.Flushable, java.lang.AutoCloseable, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanFactoryAware, org.springframework.beans.factory.InitializingBean, org.springframework.context.Lifecycle, org.springframework.context.Phased, org.springframework.context.SmartLifecycle, DataStoreWriter<T>, DataWriter<T>, PartitionDataStoreWriter<T,K>

Direct Known Subclasses:

PartitionTextFileWriter

public abstract class AbstractPartitionDataStoreWriter<T,K>
extends LifecycleObjectSupport
implements PartitionDataStoreWriter<T,K>

Base implementation of PartitionDataStoreWriter.

Author:

Janne Valkealahti

Constructor Summary

Constructors

Constructor and Description
AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec, PartitionStrategy<T,K> partitionStrategy) Instantiates a new abstract data store partition writer.

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
protected abstract DataStoreWriter<T>	createWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec) Need to be implemented by a subclass for an actual writer.
protected void	destroyWriter(org.apache.hadoop.fs.Path path) Destroys a writer with a given Path if exist.
protected void	doStart() Subclasses may implement this method with the start behaviour.
protected void	doStop() Subclasses may implement this method with the stop behaviour.
void	flush()
org.apache.hadoop.fs.Path	getBasePath() Gets the base path.
long	getCloseTimeout() Gets the close timeout.
CodecInfo	getCodec() Gets the codec.
org.apache.hadoop.conf.Configuration	getConfiguration() Gets the hadoop configuration.
FileNamingStrategyFactory<FileNamingStrategy>	getFileNamingStrategyFactory() Gets the file naming strategy factory.
long	getFlushTimeout() Gets the flush timeout.
long	getIdleTimeout() Gets the idle timeout.
java.lang.String	getInWritingPrefix() Gets the in writing prefix.
java.lang.String	getInWritingSuffix() Gets the in writing suffix.
int	getMaxOpenAttempts() Gets the max open attempts.
RolloverStrategyFactory<RolloverStrategy>	getRolloverStrategyFactory() Gets the rollover strategy factory.
boolean	isAppendable() Checks if append is enabled.
boolean	isClosed() Checks if this writer is closed.
boolean	isOverwrite() Checks if overwrite is enabled.
boolean	isSyncable() Checks if syncable is enabled.
protected void	onInit() Subclasses may implement this for initialization logic.
void	setAppendable(boolean append) Set stream as append mode.
void	setCloseTimeout(long closeTimeout) Sets the close timeout.
void	setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory) Sets the file naming strategy factory.
void	setFlushTimeout(long flushTimeout) Sets the flush timeout.
void	setIdleTimeout(long idleTimeout) Sets the idle timeout.
void	setInWritingPrefix(java.lang.String prefix) Sets the in writing prefix.
void	setInWritingSuffix(java.lang.String suffix) Sets the in writing suffix.
void	setMaxOpenAttempts(int maxOpenAttempts) Sets the max open attempts.
void	setOverwrite(boolean overwrite) Sets the flag indicating if written files may be overwritten.
void	setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory) Sets the rollover strategy factory.
void	setSyncable(boolean syncable) Sets the syncable.
void	write(T entity) Write an entity.
void	write(T entity, K partitionKey) Write an entity with an explicit partitioning key.

Methods inherited from class org.springframework.data.hadoop.store.support.LifecycleObjectSupport

afterPropertiesSet, getBeanFactory, getPhase, getStoreEventPublisher, getTaskExecutor, getTaskScheduler, isAutoStartup, isRunning, setAutoStartup, setBeanFactory, setPhase, setStoreEventPublisher, setTaskExecutor, setTaskScheduler, start, stop, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractPartitionDataStoreWriter

public AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration,
                                org.apache.hadoop.fs.Path basePath,
                                CodecInfo codec,
                                PartitionStrategy<T,K> partitionStrategy)

Instantiates a new abstract data store partition writer.

Parameters:

configuration - the hadoop configuration

basePath - the hdfs path

codec - the compression codec info

partitionStrategy - the partition strategy

Method Detail

write

public void write(T entity)
           throws java.io.IOException

Description copied from interface: DataWriter

Write an entity.

Specified by:

write in interface DataWriter<T>

Parameters:

entity - the entity to write

Throws:

java.io.IOException - if an I/O error occurs

flush

public void flush()
           throws java.io.IOException

Specified by:

flush in interface java.io.Flushable

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

write

public void write(T entity,
         K partitionKey)
           throws java.io.IOException

Description copied from interface: PartitionDataStoreWriter

Write an entity with an explicit partitioning key.

Specified by:

write in interface PartitionDataStoreWriter<T,K>

Parameters:

entity - the entity to write

partitionKey - the partition key

Throws:

java.io.IOException - if an I/O error occurs

onInit

protected void onInit()
               throws java.lang.Exception

Description copied from class: LifecycleObjectSupport

Subclasses may implement this for initialization logic. Called during the InitializingBean phase. Implementor should always call super method not to break initialization chain.

Overrides:

onInit in class LifecycleObjectSupport

Throws:

java.lang.Exception - if error occurred during init

doStart

protected void doStart()

Description copied from class: LifecycleObjectSupport

Subclasses may implement this method with the start behaviour. This method will be invoked while holding the LifecycleObjectSupport.lifecycleLock.

Overrides:

doStart in class LifecycleObjectSupport

doStop

protected void doStop()

Description copied from class: LifecycleObjectSupport

Subclasses may implement this method with the stop behaviour. This method will be invoked while holding the LifecycleObjectSupport.lifecycleLock.

Overrides:

doStop in class LifecycleObjectSupport

setFileNamingStrategyFactory

public void setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)

Sets the file naming strategy factory.

Parameters:

fileNamingStrategyFactory - the new file naming strategy factory

getFileNamingStrategyFactory

public FileNamingStrategyFactory<FileNamingStrategy> getFileNamingStrategyFactory()

Gets the file naming strategy factory.

Returns:

the file naming strategy factory

setRolloverStrategyFactory

public void setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)

Sets the rollover strategy factory.

Parameters:

rolloverStrategyFactory - the new rollover strategy factory

getRolloverStrategyFactory

public RolloverStrategyFactory<RolloverStrategy> getRolloverStrategyFactory()

Gets the rollover strategy factory.

Returns:

the rollover strategy factory

setIdleTimeout

public void setIdleTimeout(long idleTimeout)

Sets the idle timeout.

Parameters:

idleTimeout - the new idle timeout

setCloseTimeout

public void setCloseTimeout(long closeTimeout)

Sets the close timeout.

Parameters:

closeTimeout - the new close timeout

setFlushTimeout

public void setFlushTimeout(long flushTimeout)

Sets the flush timeout.

Parameters:

flushTimeout - the new flush timeout

setInWritingSuffix

public void setInWritingSuffix(java.lang.String suffix)

Sets the in writing suffix.

Parameters:

suffix - the new in writing suffix

getInWritingSuffix

public java.lang.String getInWritingSuffix()

Gets the in writing suffix.

Returns:

the in writing suffix

setInWritingPrefix

public void setInWritingPrefix(java.lang.String prefix)

Sets the in writing prefix.

Parameters:

prefix - the new in writing prefix

getInWritingPrefix

public java.lang.String getInWritingPrefix()

Gets the in writing prefix.

Returns:

the in writing prefix

setOverwrite

public void setOverwrite(boolean overwrite)

Sets the flag indicating if written files may be overwritten. Default value is FALSE meaning StoreException is thrown if file is about to get overwritten.

Parameters:

overwrite - the new overwrite

isOverwrite

public boolean isOverwrite()

Checks if overwrite is enabled.

Returns:

true, if overwrite enabled

See Also:

setOverwrite(boolean)

isAppendable

public boolean isAppendable()

Checks if append is enabled.

Returns:

true, if append enabled

setAppendable

public void setAppendable(boolean append)

Set stream as append mode.

Parameters:

append - the append flag

isSyncable

public boolean isSyncable()

Checks if syncable is enabled.

Returns:

true, if syncable is enabled

setSyncable

public void setSyncable(boolean syncable)

Sets the syncable. Enabling will result automatic call to hdfs Syncable hflush() if available. This is disable by default.

Parameters:

syncable - the syncable flag

getIdleTimeout

public long getIdleTimeout()

Gets the idle timeout.

Returns:

the idle timeout

getCloseTimeout

public long getCloseTimeout()

Gets the close timeout.

Returns:

the close timeout

getFlushTimeout

public long getFlushTimeout()

Gets the flush timeout.

Returns:

the flush timeout

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

Gets the hadoop configuration.

Returns:

the configuration

getBasePath

public org.apache.hadoop.fs.Path getBasePath()

Gets the base path.

Returns:

the base path

getCodec

public CodecInfo getCodec()

Gets the codec.

Returns:

the codec

setMaxOpenAttempts

public void setMaxOpenAttempts(int maxOpenAttempts)

Sets the max open attempts.

Parameters:

maxOpenAttempts - the new max open attempts

getMaxOpenAttempts

public int getMaxOpenAttempts()

Gets the max open attempts.

Returns:

the max open attempts

isClosed

public boolean isClosed()

Checks if this writer is closed.

Returns:

true, if is closed

createWriter

protected abstract DataStoreWriter<T> createWriter(org.apache.hadoop.conf.Configuration configuration,
                              org.apache.hadoop.fs.Path basePath,
                              CodecInfo codec)

Need to be implemented by a subclass for an actual writer.

Parameters:

configuration - the configuration

basePath - the base path

codec - the codec

Returns:

the data store writer

destroyWriter

protected void destroyWriter(org.apache.hadoop.fs.Path path)

Destroys a writer with a given Path if exist. This method expects subclass to close and flush writer before call of this.

Parameters:

path - the path