T
- the type of an entity to writeK
- the type of a partition keypublic abstract class AbstractPartitionDataStoreWriter<T,K> extends LifecycleObjectSupport implements PartitionDataStoreWriter<T,K>
PartitionDataStoreWriter
.Constructor and Description |
---|
AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration,
org.apache.hadoop.fs.Path basePath,
CodecInfo codec,
PartitionStrategy<T,K> partitionStrategy)
Instantiates a new abstract data store partition writer.
|
Modifier and Type | Method and Description |
---|---|
void |
close() |
protected abstract DataStoreWriter<T> |
createWriter(org.apache.hadoop.conf.Configuration configuration,
org.apache.hadoop.fs.Path basePath,
CodecInfo codec)
Need to be implemented by a subclass for an actual writer.
|
protected void |
doStart()
Subclasses may implement this method with the start behaviour.
|
protected void |
doStop()
Subclasses may implement this method with the stop behaviour.
|
void |
flush() |
org.apache.hadoop.fs.Path |
getBasePath()
Gets the base path.
|
CodecInfo |
getCodec()
Gets the codec.
|
org.apache.hadoop.conf.Configuration |
getConfiguration()
Gets the hadoop configuration.
|
FileNamingStrategyFactory<FileNamingStrategy> |
getFileNamingStrategyFactory()
Gets the file naming strategy factory.
|
long |
getIdleTimeout()
Gets the idle timeout.
|
java.lang.String |
getInWritingPrefix()
Gets the in writing prefix.
|
java.lang.String |
getInWritingSuffix()
Gets the in writing suffix.
|
int |
getMaxOpenAttempts()
Gets the max open attempts.
|
RolloverStrategyFactory<RolloverStrategy> |
getRolloverStrategyFactory()
Gets the rollover strategy factory.
|
boolean |
isOverwrite() |
protected void |
onInit()
Subclasses may implement this for initialization logic.
|
void |
setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
Sets the file naming strategy factory.
|
void |
setIdleTimeout(long idleTimeout)
Sets the idle timeout.
|
void |
setInWritingPrefix(java.lang.String prefix)
Sets the in writing prefix.
|
void |
setInWritingSuffix(java.lang.String suffix)
Sets the in writing suffix.
|
void |
setMaxOpenAttempts(int maxOpenAttempts)
Sets the max open attempts.
|
void |
setOverwrite(boolean overwrite)
Sets the flag indicating if written files may be overwritten.
|
void |
setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
Sets the rollover strategy factory.
|
void |
write(T entity)
Write an entity.
|
void |
write(T entity,
K partitionKey)
Write an entity with an explicit partitioning key.
|
afterPropertiesSet, getBeanFactory, getPhase, getStoreEventPublisher, getTaskExecutor, getTaskScheduler, isAutoStartup, isRunning, setAutoStartup, setBeanFactory, setPhase, setStoreEventPublisher, setTaskExecutor, setTaskScheduler, start, stop, stop
public AbstractPartitionDataStoreWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec, PartitionStrategy<T,K> partitionStrategy)
configuration
- the hadoop configurationbasePath
- the hdfs pathcodec
- the compression codec infopartitionStrategy
- the partition strategypublic void write(T entity) throws java.io.IOException
DataWriter
write
in interface DataWriter<T>
java.io.IOException
- if an I/O error occurspublic void flush() throws java.io.IOException
flush
in interface java.io.Flushable
java.io.IOException
public void close() throws java.io.IOException
close
in interface java.io.Closeable
close
in interface java.lang.AutoCloseable
java.io.IOException
public void write(T entity, K partitionKey) throws java.io.IOException
PartitionDataStoreWriter
write
in interface PartitionDataStoreWriter<T,K>
entity
- the entity to writepartitionKey
- the partition keyjava.io.IOException
- if an I/O error occursprotected void onInit() throws java.lang.Exception
LifecycleObjectSupport
InitializingBean
phase.
Implementor should always call super method not to break initialization chain.onInit
in class LifecycleObjectSupport
java.lang.Exception
protected void doStart()
LifecycleObjectSupport
LifecycleObjectSupport.lifecycleLock
.doStart
in class LifecycleObjectSupport
protected void doStop()
LifecycleObjectSupport
LifecycleObjectSupport.lifecycleLock
.doStop
in class LifecycleObjectSupport
public void setFileNamingStrategyFactory(FileNamingStrategyFactory<FileNamingStrategy> fileNamingStrategyFactory)
fileNamingStrategyFactory
- the new file naming strategy factorypublic FileNamingStrategyFactory<FileNamingStrategy> getFileNamingStrategyFactory()
public void setRolloverStrategyFactory(RolloverStrategyFactory<RolloverStrategy> rolloverStrategyFactory)
rolloverStrategyFactory
- the new rollover strategy factorypublic RolloverStrategyFactory<RolloverStrategy> getRolloverStrategyFactory()
public void setIdleTimeout(long idleTimeout)
idleTimeout
- the new idle timeoutpublic void setInWritingSuffix(java.lang.String suffix)
suffix
- the new in writing suffixpublic java.lang.String getInWritingSuffix()
public void setInWritingPrefix(java.lang.String prefix)
prefix
- the new in writing prefixpublic java.lang.String getInWritingPrefix()
public void setOverwrite(boolean overwrite)
FALSE
meaning StoreException
is thrown if file is about to get overwritten.overwrite
- the new overwritepublic boolean isOverwrite()
public long getIdleTimeout()
public org.apache.hadoop.conf.Configuration getConfiguration()
public org.apache.hadoop.fs.Path getBasePath()
public CodecInfo getCodec()
public void setMaxOpenAttempts(int maxOpenAttempts)
maxOpenAttempts
- the new max open attemptspublic int getMaxOpenAttempts()
protected abstract DataStoreWriter<T> createWriter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path basePath, CodecInfo codec)
configuration
- the configurationbasePath
- the base pathcodec
- the codec