12. File Support

12.1 Introduction

Spring Integration's File support extends the Spring Integration Core with a dedicated vocabulary to deal with reading, writing, and transforming files. It provides a namespace that enables elements defining Channel Adapters dedicated to files and support for Transformers that can read file contents into strings or byte arrays.

This section will explain the workings of FileReadingMessageSource and FileWritingMessageHandler and how to configure them as beans. Also the support for dealing with files through file specific implementations of Transformer will be discussed. Finally the file specific namespace will be explained.

12.2 Reading Files

A FileReadingMessageSource can be used to consume files from the filesystem. This is an implementation of MessageSource that creates messages from a file system directory.

<bean id="pollableFileSource"
      class="org.springframework.integration.file.FileReadingMessageSource"
        p:inputDirectory="${input.directory}"/>

To prevent creating messages for certain files, you may supply a FileListFilter. By default, an AcceptOnceFileListFilter is used. This filter ensures files are picked up only once from the directory.

<bean id="pollableFileSource"
      class="org.springframework.integration.file.FileReadingMessageSource"
        p:inputDirectory="${input.directory}"
        p:filter-ref="customFilterBean"/>

A common problem with reading files is that a file may be detected before it is ready. The default AcceptOnceFileListFilter does not prevent this. In most cases, this can be prevented if the file-writing process renames each file as soon as it is ready for reading. A filename-pattern or filename-regex filter that accepts only files that are ready (e.g. based on a known suffix), composed with the default AcceptOnceFileListFilter allows for this. The CompositeFileListFilter enables the composition.

<bean id="pollableFileSource"
      class="org.springframework.integration.file.FileReadingMessageSource"
        p:inputDirectory="${input.directory}"
        p:filter-ref="compositeFilter"/>
<bean id="compositeFilter"
      class="org.springframework.integration.file.filters.CompositeFileListFilter">
  <constructor-arg>
    <list>
      <bean class="org.springframework.integration.file.filters.AcceptOnceFileListFilter"/>
      <bean class="org.springframework.integration.file.filters.RegexPatternFileListFilter">
        <constructor-arg value="^test.*$"/>
      </bean>
    </list>
  </constructor-arg>
</bean>

The configuration can be simplified using the file specific namespace. To do this use the following template.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:int="http://www.springframework.org/schema/integration"
  xmlns:int-file="http://www.springframework.org/schema/integration/file"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/integration
    http://www.springframework.org/schema/integration/spring-integration.xsd
    http://www.springframework.org/schema/integration/file
    http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">
</beans>

Within this namespace you can reduce the FileReadingMessageSource and wrap it in an inbound Channel Adapter like this:

 <int-file:inbound-channel-adapter id="filesIn1"
           directory="file:${input.directory}" prevent-duplicates="true"/>

 <int-file:inbound-channel-adapter id="filesIn2"
           directory="file:${input.directory}"
           filter="customFilterBean" />

 <int-file:inbound-channel-adapter id="filesIn3"
           directory="file:${input.directory}"
           filename-pattern="test*" />

 <int-file:inbound-channel-adapter id="filesIn4"
           directory="file:${input.directory}"
           filename-regex="test[0-9]+\.txt" />              

The first channel adapter is relying on the default filter that just prevents duplication, the second is using a custom filter, the third is using the filename-pattern attribute to add an AntPathMatcher based filter, and the fourth is using the filename-regex attribute to add a regular expression Pattern based filter to the FileReadingMessageSource. The filename-pattern and filename-regex attributes are each mutually exclusive with the regular filter reference attribute. However, you can use the filter attribute to reference an instance of CompositeFileListFilter that combines any number of filters, including one or more pattern based filters to fit your particular needs.

When multiple processes are reading from the same directory it can be desirable to lock files to prevent them from being picked up concurrently. To do this you can use a FileLocker. There is a java.nio based implementation available out of the box, but it is also possible to implement your own locking scheme. The nio locker can be injected as follows

 <int-file:inbound-channel-adapter id="filesIn"
         directory="file:${input.directory}" prevent-duplicates="true">
     <int-file:nio-locker/>
 </int-file:inbound-channel-adapter>

A custom locker you can configure like this:

 <int-file:inbound-channel-adapter id="filesIn"
         directory="file:${input.directory}" prevent-duplicates="true">
     <int-file:locker ref="customLocker"/>
 </int-file:inbound-channel-adapter>

[Note]Note
When a file inbound adapter is configured with a locker, it will take the responsibility to acquire a lock before the file is allowed to be received. It will not assume the responsibility to unlock the file. If you have processed the file and keeping the locks hanging around you have a memory leak. If this is a problem in your case you should call FileLocker.unlock(File file) yourself at the appropriate time.

When filtering and locking files is not enough it might be needed to control the way files are listed entirely. To implement this type of requirement you can use an implementation of DirectoryScanner. This scanner allows you to determine entirely what files are listed each poll. This is also the interface that Spring Integration uses internally to wire FileListFilters FileLocker to the FileReadingMessageSource. A custom DirectoryScanner can be injected into the <int-file:inbound-channel-adapter/> on the scanner attribute.

 <int-file:inbound-channel-adapter id="filesIn" directory="file:${input.directory}"
       prevent-duplicates="true" scanner="customDirectoryScanner"/>

This gives you full freedom to choose the ordering, listing and locking strategies.

12.3 Writing files

To write messages to the file system you can use a FileWritingMessageHandler. This class can deal with File, String, or byte array payloads. In its simplest form the FileWritingMessageHandler only requires a destination directory for writing the files. The name of the file to be written is determined by the handler's FileNameGenerator. The default implementation looks for a Message header whose key matches the constant defined as FileHeaders.FILENAME.

Additionally, you can configure the encoding and the charset that will be used in case of a String payload.

To make things easier, you can configure the FileWritingMessageHandler as part of an Outbound Channel Adapter or Outbound Gateway using the provided XML namespace support.

12.3.1 Specifying the Output Directory

Both, the File Oubound Channel Adapter and the File Outbound Gateway provide two configuration attributes for specifying the output directory:

  • directory

  • directory-expression

[Note]Note
The directory-expression attribute is available since Spring Integration 2.2.

Using the directory attribute

When using the directory attribute, the output directory will be set to a fixed value, that is set at intialization time of the FileWritingMessageHandler. If you don't specify this attribute, then you must use the directory-expression attribute.

Using the directory-expression attribute

If you want to have full SpEL support you would choose the directory-expression attribute. This attribute accepts a SpEL expression that is evaluated for each message being processed. Thus, you have full access to a Message's payload and its headers to dynamically specify the output file directory.

The SpEL expression must resolve to either a String or to java.io.File. Furthermore the resulting String or File must point to a directory. If you don't specify the directory-expression attribute, then you must set the directory attribute.

Using the auto-create-directory attribute

If the destination directory does not exists, yet, by default the respective destination directory and any non-existing parent directories are being created automatically. You can set the auto-create-directory attribute to false in order to prevent that. This attribute applies to both, the directory and the directory-expression attribute.

[Note]Note

When using the directory attribute and auto-create-directory is false, the following change was made starting with Spring Integration 2.2:

Instead of checking for the existence of the destination directory at initialization time of the adapter, this check is now performed for each message being processed.

Furthermore, if auto-create-directory is true and the directory was deleted between the processing of messages, the directory will be re-created for each message being processed.

12.3.2 Dealing with Existing Destination Files

When writing files and the destination file already exists, the default behavior is to overwrite that target file. This behavior, though, can be changed by setting the mode attribute on the respective File Outbound components. The following options exist:

  • REPLACE (Default)
  • APPEND
  • FAIL
  • IGNORE
[Note]Note
The mode attribute and the options APPEND, FAIL and IGNORE, are available since Spring Integration 2.2.

REPLACE

If the target file already exists, it will be overwritten. If the mode attribute is not specified, then this is the default behavior when writing files.

APPEND

This mode allows you to append Message content to the existing file instead of creating a new file each time. Note that this attribute is mutually exclusive with temporary-file-suffix attribute since when appending content to the existing file, the adapter no longer uses a temporary file.

FAIL

If the target file exists, a MessageHandlingException is thrown.

IGNORE

If the target file exists, the message payload is silently ignored.

12.3.3 File Outbound Channel Adapter

 <int-file:outbound-channel-adapter id="filesOut" directory="${input.directory.property}"/>

The namespace based configuration also supports a delete-source-files attribute. If set to true, it will trigger the deletion of the original source files after writing to a destination. The default value for that flag is false.

 <int-file:outbound-channel-adapter id="filesOut"
    directory="${output.directory}"
    delete-source-files="true"/>
[Note]Note
The delete-source-files attribute will only have an effect if the inbound Message has a File payload or if the FileHeaders.ORIGINAL_FILE header value contains either the source File instance or a String representing the original file path.

12.3.4 Outbound Gateway

In cases where you want to continue processing messages based on the written file, you can use the outbound-gateway instead. It plays a very similar role as the outbound-channel-adapter. However, after writing the file, it will also send it to the reply channel as the payload of a Message.

<int-file:outbound-gateway id="mover" request-channel="moveInput"
    reply-channel="output"
    directory="${output.directory}"
    mode="REPLACE" delete-source-files="true"/>

As mentioned earlier, you can also specify the mode attribute, which defines the behavior of how to deal with situations where the destination file already exists. Please see Section 12.3.2, “Dealing with Existing Destination Files” for further details. Generally, when using the File Outbound Gateway, the result file is returned as the Message payload on the reply channel.

This also applies when specifying the IGNORE mode. In that case the pre-existing destination file is returned. If the payload of the request message was a file, you still have access to that original file through the Message Header FileHeaders.ORIGINAL_FILE.

[Note]Note
The 'outbound-gateway' works well in cases where you want to first move a file and then send it through a processing pipeline. In such cases, you may connect the file namespace's 'inbound-channel-adapter' element to the 'outbound-gateway' and then connect that gateway's reply-channel to the beginning of the pipeline.

If you have more elaborate requirements or need to support additional payload types as input to be converted to file content you could extend the FileWritingMessageHandler, but a much better option is to rely on a Transformer.

12.4 File Transformers

To transform data read from the file system to objects and the other way around you need to do some work. Contrary to FileReadingMessageSource and to a lesser extent FileWritingMessageHandler, it is very likely that you will need your own mechanism to get the job done. For this you can implement the Transformer interface. Or extend the AbstractFilePayloadTransformer for inbound messages. Some obvious implementations have been provided.

FileToByteArrayTransformer transforms Files into byte[]s using Spring's FileCopyUtils. It is often better to use a sequence of transformers than to put all transformations in a single class. In that case the File to byte[] conversion might be a logical first step.

FileToStringTransformer will convert Files to Strings as the name suggests. If nothing else, this can be useful for debugging (consider using with a Wire Tap).

To configure File specific transformers you can use the appropriate elements from the file namespace.

 <int-file:file-to-bytes-transformer  input-channel="input" output-channel="output"
             delete-files="true"/>

 <int-file:file-to-string-transformer input-channel="input" output-channel="output"
             delete-files="true" charset="UTF-8"/>

The delete-files option signals to the transformer that it should delete the inbound File after the transformation is complete. This is in no way a replacement for using the AcceptOnceFileListFilter when the FileReadingMessageSource is being used in a multi-threaded environment (e.g. Spring Integration in general).