T
- the type of an entity to writeK
- the type of a partition keypublic abstract class AbstractPartitionDataStoreWriter<T,K> extends LifecycleObjectSupport implements PartitionDataStoreWriter<T,K>
PartitionDataStoreWriter
.Constructor and Description |
---|
AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration,
org.apache.hadoop.fs.Path basePath,
CodecInfo codec,
PartitionStrategy<T,K> partitionStrategy)
Instantiates a new abstract data store partition writer.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
protected abstract DataStoreWriter<T> |
createWriter(org.apache.hadoop.conf.Configuration configuration,
org.apache.hadoop.fs.Path basePath,
CodecInfo codec)
Need to be implemented by a subclass for an actual writer.
|
protected void |
destroyWriter(org.apache.hadoop.fs.Path path)
Destroys a writer with a given
Path if exist. |
protected void |
doStart()
Subclasses may implement this method with the start behaviour.
|
protected void |
doStop()
Subclasses may implement this method with the stop behaviour.
|
void |
flush() |
org.apache.hadoop.fs.Path |
getBasePath()
Gets the base path.
|
long |
getCloseTimeout()
Gets the close timeout.
|
CodecInfo |
getCodec()
Gets the codec.
|
org.apache.hadoop.conf.Configuration |
getConfiguration()
Gets the hadoop configuration.
|
FileNamingStrategyFactory<FileNamingStrategy> |
getFileNamingStrategyFactory()
Gets the file naming strategy factory.
|
long |
getFlushTimeout()
Gets the flush timeout.
|
long |
getIdleTimeout()
Gets the idle timeout.
|
java.lang.String |
getInWritingPrefix()
Gets the in writing prefix.
|
java.lang.String |
getInWritingSuffix()
Gets the in writing suffix.
|
int |
getMaxOpenAttempts()
Gets the max open attempts.
|
RolloverStrategyFactory<RolloverStrategy> |
getRolloverStrategyFactory()
Gets the rollover strategy factory.
|
boolean |
isAppendable()
Checks if append is enabled.
|
boolean |
isClosed()
Checks if this writer is closed.
|
boolean |
isOverwrite()
Checks if overwrite is enabled.
|
boolean |
isSyncable()
Checks if syncable is enabled.
|
protected void |
onInit()
Subclasses may implement this for initialization logic.
|
void |
setAppendable(boolean append)
Set stream as append mode.
|
void |
setCloseTimeout(long closeTimeout)
Sets the close timeout.
|
void |
setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
Sets the file naming strategy factory.
|
void |
setFlushTimeout(long flushTimeout)
Sets the flush timeout.
|
void |
setIdleTimeout(long idleTimeout)
Sets the idle timeout.
|
void |
setInWritingPrefix(java.lang.String prefix)
Sets the in writing prefix.
|
void |
setInWritingSuffix(java.lang.String suffix)
Sets the in writing suffix.
|
void |
setMaxOpenAttempts(int maxOpenAttempts)
Sets the max open attempts.
|
void |
setOverwrite(boolean overwrite)
Sets the flag indicating if written files may be overwritten.
|
void |
setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
Sets the rollover strategy factory.
|
void |
setSyncable(boolean syncable)
Sets the syncable.
|
void |
write(T entity)
Write an entity.
|
void |
write(T entity,
K partitionKey)
Write an entity with an explicit partitioning key.
|
afterPropertiesSet, getBeanFactory, getPhase, getStoreEventPublisher, getTaskExecutor, getTaskScheduler, isAutoStartup, isRunning, setAutoStartup, setBeanFactory, setPhase, setStoreEventPublisher, setTaskExecutor, setTaskScheduler, start, stop, stop
public AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec, PartitionStrategy<T,K> partitionStrategy)
configuration
- the hadoop configurationbasePath
- the hdfs pathcodec
- the compression codec infopartitionStrategy
- the partition strategypublic void write(T entity) throws java.io.IOException
DataWriter
write
in interface DataWriter<T>
entity
- the entity to writejava.io.IOException
- if an I/O error occurspublic void flush() throws java.io.IOException
flush
in interface java.io.Flushable
java.io.IOException
public void close() throws java.io.IOException
close
in interface java.io.Closeable
close
in interface java.lang.AutoCloseable
java.io.IOException
public void write(T entity, K partitionKey) throws java.io.IOException
PartitionDataStoreWriter
write
in interface PartitionDataStoreWriter<T,K>
entity
- the entity to writepartitionKey
- the partition keyjava.io.IOException
- if an I/O error occursprotected void onInit() throws java.lang.Exception
LifecycleObjectSupport
InitializingBean
phase.
Implementor should always call super method not to break initialization chain.onInit
in class LifecycleObjectSupport
java.lang.Exception
- if error occurred during initprotected void doStart()
LifecycleObjectSupport
LifecycleObjectSupport.lifecycleLock
.doStart
in class LifecycleObjectSupport
protected void doStop()
LifecycleObjectSupport
LifecycleObjectSupport.lifecycleLock
.doStop
in class LifecycleObjectSupport
public void setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
fileNamingStrategyFactory
- the new file naming strategy factorypublic FileNamingStrategyFactory<FileNamingStrategy> getFileNamingStrategyFactory()
public void setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
rolloverStrategyFactory
- the new rollover strategy factorypublic RolloverStrategyFactory<RolloverStrategy> getRolloverStrategyFactory()
public void setIdleTimeout(long idleTimeout)
idleTimeout
- the new idle timeoutpublic void setCloseTimeout(long closeTimeout)
closeTimeout
- the new close timeoutpublic void setFlushTimeout(long flushTimeout)
flushTimeout
- the new flush timeoutpublic void setInWritingSuffix(java.lang.String suffix)
suffix
- the new in writing suffixpublic java.lang.String getInWritingSuffix()
public void setInWritingPrefix(java.lang.String prefix)
prefix
- the new in writing prefixpublic java.lang.String getInWritingPrefix()
public void setOverwrite(boolean overwrite)
FALSE
meaning StoreException
is thrown if file is about to get overwritten.overwrite
- the new overwritepublic boolean isOverwrite()
setOverwrite(boolean)
public boolean isAppendable()
public void setAppendable(boolean append)
append
- the append flagpublic boolean isSyncable()
public void setSyncable(boolean syncable)
syncable
- the syncable flagpublic long getIdleTimeout()
public long getCloseTimeout()
public long getFlushTimeout()
public org.apache.hadoop.conf.Configuration getConfiguration()
public org.apache.hadoop.fs.Path getBasePath()
public CodecInfo getCodec()
public void setMaxOpenAttempts(int maxOpenAttempts)
maxOpenAttempts
- the new max open attemptspublic int getMaxOpenAttempts()
public boolean isClosed()
protected abstract DataStoreWriter<T> createWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec)
configuration
- the configurationbasePath
- the base pathcodec
- the codecprotected void destroyWriter(org.apache.hadoop.fs.Path path)
Path
if exist.
This method expects subclass to close and flush writer
before call of this.path
- the path