As with RowMapper, which takes a low-level construct such as ResultSet and returns an Object, flat file processing requires the same construct to convert a String line into an Object, as shown in the following interface definition:

The basic contract is that, given the current line and the line number with which it is associated, the mapper should return a resulting domain object. This is similar to RowMapper, in that each line is associated with its line number, just as each row in a ResultSet is tied to its row number. This allows the line number to be tied to the resulting domain object for identity comparison or for more informative logging. However, unlike RowMapper, the LineMapper is given a raw line which, as discussed above, only gets you halfway there. The line must be tokenized into a FieldSet, which can then be mapped to an object, as described later in this document.

An abstraction for turning a line of input into a FieldSet is necessary because there can be many formats of flat file data that need to be converted to a FieldSet. In Spring Batch, this interface is the LineTokenizer:

The contract of a LineTokenizer is such that, given a line of input (in theory the String could encompass more than one line), a FieldSet representing the line is returned. This FieldSet can then be passed to a FieldSetMapper. Spring Batch contains the following LineTokenizer implementations:

The FieldSetMapper interface defines a single method, mapFieldSet, which takes a FieldSet object and maps its contents to an object. This object may be a custom DTO, a domain object, or an array, depending on the needs of the job. The FieldSetMapper is used in conjunction with the LineTokenizer to translate a line of data from a resource into an object of the desired type, as shown in the following interface definition:

The pattern used is the same as the RowMapper used by JdbcTemplate.

Now that the basic interfaces for reading in flat files have been defined, it becomes clear that three basic steps are required:

The two interfaces described above represent two separate tasks: converting a line into a FieldSet and mapping a FieldSet to a domain object. Because the input of a LineTokenizer matches the input of the LineMapper (a line), and the output of a FieldSetMapper matches the output of the LineMapper, a default implementation that uses both a LineTokenizer and a FieldSetMapper is provided. The DefaultLineMapper, shown in the following class definition, represents the behavior most users need:

The above functionality is provided in a default implementation, rather than being built into the reader itself (as was done in previous versions of the framework) to allow users greater flexibility in controlling the parsing process, especially if access to the raw line is needed.

The following example illustrates how to read a flat file with an actual domain scenario. This particular batch job reads in football players from the following file:

The contents of this file are mapped to the following Player domain object:

To map a FieldSet into a Player object, a FieldSetMapper that returns players needs to be defined, as shown in the following example:

The file can then be read by correctly constructing a FlatFileItemReader and calling read, as shown in the following example:

Each call to read returns a new Player object from each line in the file. When the end of the file is reached, null is returned.

There is one additional piece of functionality that is allowed by both DelimitedLineTokenizer and FixedLengthTokenizer and that is similar in function to a JDBC ResultSet. The names of the fields can be injected into either of these LineTokenizer implementations to increase the readability of the mapping function. First, the column names of all fields in the flat file are injected into the tokenizer, as shown in the following example:

A FieldSetMapper can use this information as follows:

For many, having to write a specific FieldSetMapper is equally as cumbersome as writing a specific RowMapper for a JdbcTemplate. Spring Batch makes this easier by providing a FieldSetMapper that automatically maps fields by matching a field name with a setter on the object using the JavaBean specification.

For each entry in the FieldSet, the mapper looks for a corresponding setter on a new instance of the Player object (for this reason, prototype scope is required) in the same way the Spring container looks for setters matching a property name. Each available field in the FieldSet is mapped, and the resultant Player object is returned, with no code required.

So far, only delimited files have been discussed in much detail. However, they represent only half of the file reading picture. Many organizations that use flat files use fixed length formats. An example fixed length file follows:

While this looks like one large field, it actually represent 4 distinct fields:

When configuring the FixedLengthLineTokenizer, each of these lengths must be provided in the form of ranges.

Because the FixedLengthLineTokenizer uses the same LineTokenizer interface as discussed above, it returns the same FieldSet as if a delimiter had been used. This lets the same approaches be used in handling its output, such as using the BeanWrapperFieldSetMapper.

All of the file reading examples up to this point have all made a key assumption for simplicity’s sake: all of the records in a file have the same format. However, this may not always be the case. It is very common that a file might have records with different formats that need to be tokenized differently and mapped to different objects. The following excerpt from a file illustrates this:

In this file we have three types of records, "USER", "LINEA", and "LINEB". A "USER" line corresponds to a User object. "LINEA" and "LINEB" both correspond to Line objects, though a "LINEA" has more information than a "LINEB".

The ItemReader reads each line individually, but we must specify different LineTokenizer and FieldSetMapper objects so that the ItemWriter receives the correct items. The PatternMatchingCompositeLineMapper makes this easy by allowing maps of patterns to LineTokenizers and patterns to FieldSetMappers to be configured.

In this example, "LINEA" and "LINEB" have separate LineTokenizer instances, but they both use the same FieldSetMapper.

The PatternMatchingCompositeLineMapper uses the PatternMatcher#match method in order to select the correct delegate for each line. The PatternMatcher allows for two wildcard characters with special meaning: the question mark ("?") matches exactly one character, while the asterisk ("*") matches zero or more characters. Note that, in the preceding configuration, all patterns end with an asterisk, making them effectively prefixes to lines. The PatternMatcher always matches the most specific pattern possible, regardless of the order in the configuration. So if "LINE*" and "LINEA*" were both listed as patterns, "LINEA" would match pattern "LINEA*", while "LINEB" would match pattern "LINE*". Additionally, a single asterisk ("*") can serve as a default by matching any line not matched by any other pattern.

There is also a PatternMatchingCompositeLineTokenizer that can be used for tokenization alone.

It is also common for a flat file to contain records that each span multiple lines. To handle this situation, a more complex strategy is required. A demonstration of this common pattern can be found in the multiLineRecords sample.

There are many scenarios when tokenizing a line may cause exceptions to be thrown. Many flat files are imperfect and contain incorrectly formatted records. Many users choose to skip these erroneous lines while logging the issue, the original line, and the line number. These logs can later be inspected manually or by another batch job. For this reason, Spring Batch provides a hierarchy of exceptions for handling parse exceptions: FlatFileParseException and FlatFileFormatException. FlatFileParseException is thrown by the FlatFileItemReader when any errors are encountered while trying to read a file. FlatFileFormatException is thrown by implementations of the LineTokenizer interface and indicates a more specific error encountered while tokenizing.

Just as the LineTokenizer interface is necessary to take an item and turn it into a String, file writing must have a way to aggregate multiple fields into a single string for writing to a file. In Spring Batch, this is the LineAggregator, shown in the following interface definition:

The LineAggregator is the logical opposite of LineTokenizer. LineTokenizer takes a String and returns a FieldSet, whereas LineAggregator takes an item and returns a String.

Now that the LineAggregator interface and its most basic implementation, PassThroughLineAggregator, have been defined, the basic flow of writing can be explained:

The following excerpt from the FlatFileItemWriter expresses this in code:

The preceding example may be useful for the most basic uses of a writing to a file. However, most users of the FlatFileItemWriter have a domain object that needs to be written out and, thus, must be converted into a line. In file reading, the following was required:

File writing has similar but inverse steps:

Because there is no way for the framework to know which fields from the object need to be written out, a FieldExtractor must be written to accomplish the task of turning the item into an array, as shown in the following interface definition:

Implementations of the FieldExtractor interface should create an array from the fields of the provided object, which can then be written out with a delimiter between the elements or as part of a fixed-width line.

The most basic flat file format is one in which all fields are separated by a delimiter. This can be accomplished using a DelimitedLineAggregator. The following example writes out a simple domain object that represents a credit to a customer account:

Because a domain object is being used, an implementation of the FieldExtractor interface must be provided, along with the delimiter to use.

In the previous example, the BeanWrapperFieldExtractor described earlier in this chapter is used to turn the name and credit fields within CustomerCredit into an object array, which is then written out with commas between each field.

It is also possible to use the FlatFileItemWriterBuilder.DelimitedBuilder to automatically create the BeanWrapperFieldExtractor and DelimitedLineAggregator as shown in the following example:

Delimited is not the only type of flat file format. Many prefer to use a set width for each column to delineate between fields, which is usually referred to as 'fixed width'. Spring Batch supports this in file writing with the FormatterLineAggregator.

Most of the preceding example should look familiar. However, the value of the format property is new.

The underlying implementation is built using the same Formatter added as part of Java 5. The Java Formatter is based on the printf functionality of the C programming language. Most details on how to configure a formatter can be found in the Javadoc of Formatter.

It is also possible to use the FlatFileItemWriterBuilder.FormattedBuilder to automatically create the BeanWrapperFieldExtractor and FormatterLineAggregator as shown in following example:

FlatFileItemReader has a very simple relationship with file resources. When the reader is initialized, it opens the file (if it exists), and throws an exception if it does not. File writing isn’t quite so simple. At first glance, it seems like a similar straightforward contract should exist for FlatFileItemWriter: If the file already exists, throw an exception, and, if it does not, create it and start writing. However, potentially restarting a Job can cause issues. In normal restart scenarios, the contract is reversed: If the file exists, start writing to it from the last known good position, and, if it does not, throw an exception. However, what happens if the file name for this job is always the same? In this case, you would want to delete the file if it exists, unless it’s a restart. Because of this possibility, the FlatFileItemWriter contains the property, shouldDeleteIfExists. Setting this property to true causes an existing file with the same name to be deleted when the writer is opened.

JdbcCursorItemReader is the JDBC implementation of the cursor-based technique. It works directly with a ResultSet and requires an SQL statement to run against a connection obtained from a DataSource. The following database schema is used as an example:

Many people prefer to use a domain object for each row, so the following example uses an implementation of the RowMapper interface to map a CustomerCredit object:

Because JdbcCursorItemReader shares key interfaces with JdbcTemplate, it is useful to see an example of how to read in this data with JdbcTemplate, in order to contrast it with the ItemReader. For the purposes of this example, assume there are 1,000 rows in the CUSTOMER database. The first example uses JdbcTemplate:

After running the preceding code snippet, the customerCredits list contains 1,000 CustomerCredit objects. In the query method, a connection is obtained from the DataSource, the provided SQL is run against it, and the mapRow method is called for each row in the ResultSet. Contrast this with the approach of the JdbcCursorItemReader, shown in the following example:

After running the preceding code snippet, the counter equals 1,000. If the code above had put the returned customerCredit into a list, the result would have been exactly the same as with the JdbcTemplate example. However, the big advantage of the ItemReader is that it allows items to be 'streamed'. The read method can be called once, the item can be written out by an ItemWriter, and then the next item can be obtained with read. This allows item reading and writing to be done in 'chunks' and committed periodically, which is the essence of high performance batch processing. Furthermore, it is easily configured for injection into a Spring Batch Step.

Just as normal Spring users make important decisions about whether or not to use ORM solutions, which affect whether or not they use a JdbcTemplate or a HibernateTemplate, Spring Batch users have the same options. HibernateCursorItemReader is the Hibernate implementation of the cursor technique. Hibernate’s usage in batch has been fairly controversial. This has largely been because Hibernate was originally developed to support online application styles. However, that does not mean it cannot be used for batch processing. The easiest approach for solving this problem is to use a StatelessSession rather than a standard session. This removes all of the caching and dirty checking Hibernate employs and that can cause issues in a batch scenario. For more information on the differences between stateless and normal hibernate sessions, refer to the documentation of your specific hibernate release. The HibernateCursorItemReader lets you declare an HQL statement and pass in a SessionFactory, which will pass back one item per call to read in the same basic fashion as the JdbcCursorItemReader. The following example configuration uses the same 'customer credit' example as the JDBC reader:

This configured ItemReader returns CustomerCredit objects in the exact same manner as described by the JdbcCursorItemReader, assuming hibernate mapping files have been created correctly for the Customer table. The 'useStatelessSession' property defaults to true but has been added here to draw attention to the ability to switch it on or off. It is also worth noting that the fetch size of the underlying cursor can be set with the setFetchSize property. As with JdbcCursorItemReader, configuration is straightforward.

Sometimes it is necessary to obtain the cursor data by using a stored procedure. The StoredProcedureItemReader works like the JdbcCursorItemReader, except that, instead of running a query to obtain a cursor, it runs a stored procedure that returns a cursor. The stored procedure can return the cursor in three different ways:

The preceding example relies on the stored procedure to provide a ResultSet as a returned result (option 1 from earlier).

If the stored procedure returned a ref-cursor (option 2), then we would need to provide the position of the out parameter that is the returned ref-cursor.

If the cursor was returned from a stored function (option 3), we would need to set the property "function" to true. It defaults to false.

In all of these cases, we need to define a RowMapper as well as a DataSource and the actual procedure name.

If the stored procedure or function takes in parameters, then they must be declared and set by using the parameters property. The following example, for Oracle, declares three parameters. The first one is the out parameter that returns the ref-cursor, and the second and third are in parameters that takes a value of type INTEGER.

In addition to the parameter declarations, we need to specify a PreparedStatementSetter implementation that sets the parameter values for the call. This works the same as for the JdbcCursorItemReader above. All the additional properties listed in Additional Properties apply to the StoredProcedureItemReader as well.

One implementation of a paging ItemReader is the JdbcPagingItemReader. The JdbcPagingItemReader needs a PagingQueryProvider responsible for providing the SQL queries used to retrieve the rows making up a page. Since each database has its own strategy for providing paging support, we need to use a different PagingQueryProvider for each supported database type. There is also the SqlPagingQueryProviderFactoryBean that auto-detects the database that is being used and determine the appropriate PagingQueryProvider implementation. This simplifies the configuration and is the recommended best practice.

The SqlPagingQueryProviderFactoryBean requires that you specify a select clause and a from clause. You can also provide an optional where clause. These clauses and the required sortKey are used to build an SQL statement.

After the reader has been opened, it passes back one item per call to read in the same basic fashion as any other ItemReader. The paging happens behind the scenes when additional rows are needed.

This configured ItemReader returns CustomerCredit objects using the RowMapper, which must be specified. The 'pageSize' property determines the number of entities read from the database for each query run.

The 'parameterValues' property can be used to specify a Map of parameter values for the query. If you use named parameters in the where clause, the key for each entry should match the name of the named parameter. If you use a traditional '?' placeholder, then the key for each entry should be the number of the placeholder, starting with 1.

Another implementation of a paging ItemReader is the JpaPagingItemReader. JPA does not have a concept similar to the Hibernate StatelessSession, so we have to use other features provided by the JPA specification. Since JPA supports paging, this is a natural choice when it comes to using JPA for batch processing. After each page is read, the entities become detached and the persistence context is cleared, to allow the entities to be garbage collected once the page is processed.

The JpaPagingItemReader lets you declare a JPQL statement and pass in a EntityManagerFactory. It then passes back one item per call to read in the same basic fashion as any other ItemReader. The paging happens behind the scenes when additional entities are needed.

This configured ItemReader returns CustomerCredit objects in the exact same manner as described for the JdbcPagingItemReader above, assuming the CustomerCredit object has the correct JPA annotations or ORM mapping file. The 'pageSize' property determines the number of entities read from the database for each query execution.

The final challenge is to make the ItemReader restartable. Currently, if processing is interrupted and begins again, the ItemReader must start at the beginning. This is actually valid in many scenarios, but it is sometimes preferable that a batch job restarts where it left off. The key discriminant is often whether the reader is stateful or stateless. A stateless reader does not need to worry about restartability, but a stateful one has to try to reconstitute its last known state on restart. For this reason, we recommend that you keep custom readers stateless if possible, so you need not worry about restartability.

If you do need to store state, then the ItemStream interface should be used:

On each call to the ItemStream update method, the current index of the ItemReader is stored in the provided ExecutionContext with a key of 'current.index'. When the ItemStream open method is called, the ExecutionContext is checked to see if it contains an entry with that key. If the key is found, then the current index is moved to that location. This is a fairly trivial example, but it still meets the general contract:

Most ItemReaders have much more sophisticated restart logic. The JdbcCursorItemReader, for example, stores the row ID of the last processed row in the cursor.

It is also worth noting that the key used within the ExecutionContext should not be trivial. That is because the same ExecutionContext is used for all ItemStreams within a Step. In most cases, simply prepending the key with the class name should be enough to guarantee uniqueness. However, in the rare cases where two of the same type of ItemStream are used in the same step (which can happen if two files are needed for output), a more unique name is needed. For this reason, many of the Spring Batch ItemReader and ItemWriter implementations have a setName() property that lets this key name be overridden.

To make the ItemWriter restartable, we would follow the same process as for the ItemReader, adding and implementing the ItemStream interface to synchronize the execution context. In the example, we might have to count the number of items processed and add that as a footer record. If we needed to do that, we could implement ItemStream in our ItemWriter so that the counter was reconstituted from the execution context if the stream was re-opened.

In many realistic cases, custom ItemWriters also delegate to another writer that itself is restartable (for example, when writing to a file), or else it writes to a transactional resource and so does not need to be restartable, because it is stateless. When you have a stateful writer you should probably be sure to implement ItemStream as well as ItemWriter. Remember also that the client of the writer needs to be aware of the ItemStream, so you may need to register it as a stream in the configuration.

When using an ItemReader that is not thread safe, Spring Batch offers the SynchronizedItemStreamReader decorator, which can be used to make the ItemReader thread safe. Spring Batch provides a SynchronizedItemStreamReaderBuilder to construct an instance of the SynchronizedItemStreamReader.

Spring Batch includes a decorator that adds a peek method to an ItemReader. This peek method lets the user peek one item ahead. Repeated calls to the peek returns the same item, and this is the next item returned from the read method. Spring Batch provides a SingleItemPeekableItemReaderBuilder to construct an instance of the SingleItemPeekableItemReader.

When using an ItemWriter that is not thread safe, Spring Batch offers the SynchronizedItemStreamWriter decorator, which can be used to make the ItemWriter thread safe. Spring Batch provides a SynchronizedItemStreamWriterBuilder to construct an instance of the SynchronizedItemStreamWriter.

The MultiResourceItemWriter wraps a ResourceAwareItemWriterItemStream and creates a new output resource when the count of items written in the current resource exceeds the itemCountLimitPerResource. Spring Batch provides a MultiResourceItemWriterBuilder to construct an instance of the MultiResourceItemWriter.

The ClassifierCompositeItemWriter calls one of a collection of ItemWriter implementations for each item, based on a router pattern implemented through the provided Classifier. The implementation is thread-safe if all delegates are thread-safe. Spring Batch provides a ClassifierCompositeItemWriterBuilder to construct an instance of the ClassifierCompositeItemWriter.

The ClassifierCompositeItemProcessor is an ItemProcessor that calls one of a collection of ItemProcessor implementations, based on a router pattern implemented through the provided Classifier. Spring Batch provides a ClassifierCompositeItemProcessorBuilder to construct an instance of the ClassifierCompositeItemProcessor.

The AmqpItemReader is an ItemReader that uses an AmqpTemplate to receive or convert messages from an exchange. Spring Batch provides a AmqpItemReaderBuilder to construct an instance of the AmqpItemReader.

The AmqpItemWriter is an ItemWriter that uses an AmqpTemplate to send messages to an AMQP exchange. Messages are sent to the nameless exchange if the name not specified in the provided AmqpTemplate. Spring Batch provides an AmqpItemWriterBuilder to construct an instance of the AmqpItemWriter.

The JmsItemReader is an ItemReader for JMS that uses a JmsTemplate. The template should have a default destination, which is used to provide items for the read() method. Spring Batch provides a JmsItemReaderBuilder to construct an instance of the JmsItemReader.

The JmsItemWriter is an ItemWriter for JMS that uses a JmsTemplate. The template should have a default destination, which is used to send items in write(List). Spring Batch provides a JmsItemWriterBuilder to construct an instance of the JmsItemWriter.

The KafkaItemReader is an ItemReader for an Apache Kafka topic. It can be configured to read messages from multiple partitions of the same topic. It stores message offsets in the execution context to support restart capabilities. Spring Batch provides a KafkaItemReaderBuilder to construct an instance of the KafkaItemReader.

The KafkaItemWriter is an ItemWriter for Apache Kafka that uses a KafkaTemplate to send events to a default topic. Spring Batch provides a KafkaItemWriterBuilder to construct an instance of the KafkaItemWriter.

The Neo4jItemReader is an ItemReader that reads objects from the graph database Neo4j by using a paging technique. Spring Batch provides a Neo4jItemReaderBuilder to construct an instance of the Neo4jItemReader.

The MongoItemReader is an ItemReader that reads documents from MongoDB by using a paging technique. Spring Batch provides a MongoItemReaderBuilder to construct an instance of the MongoItemReader.

The HibernateCursorItemReader is an ItemStreamReader for reading database records built on top of Hibernate. It executes the HQL query and then, when initialized, iterates over the result set as the read() method is called, successively returning an object corresponding to the current row. Spring Batch provides a HibernateCursorItemReaderBuilder to construct an instance of the HibernateCursorItemReader.

The HibernatePagingItemReader is an ItemReader for reading database records built on top of Hibernate and reading only up to a fixed number of items at a time. Spring Batch provides a HibernatePagingItemReaderBuilder to construct an instance of the HibernatePagingItemReader.

The RepositoryItemReader is an ItemReader that reads records by using a PagingAndSortingRepository. Spring Batch provides a RepositoryItemReaderBuilder to construct an instance of the RepositoryItemReader.

The Neo4jItemWriter is an ItemWriter implementation that writes to a Neo4j database. Spring Batch provides a Neo4jItemWriterBuilder to construct an instance of the Neo4jItemWriter.

The MongoItemWriter is an ItemWriter implementation that writes to a MongoDB store using an implementation of Spring Data’s MongoOperations. Spring Batch provides a MongoItemWriterBuilder to construct an instance of the MongoItemWriter.

The RepositoryItemWriter is an ItemWriter wrapper for a CrudRepository from Spring Data. Spring Batch provides a RepositoryItemWriterBuilder to construct an instance of the RepositoryItemWriter.

The HibernateItemWriter is an ItemWriter that uses a Hibernate session to save or update entities that are not part of the current Hibernate session. Spring Batch provides a HibernateItemWriterBuilder to construct an instance of the HibernateItemWriter.

The JdbcBatchItemWriter is an ItemWriter that uses the batching features from NamedParameterJdbcTemplate to execute a batch of statements for all items provided. Spring Batch provides a JdbcBatchItemWriterBuilder to construct an instance of the JdbcBatchItemWriter.

The JpaItemWriter is an ItemWriter that uses a JPA EntityManagerFactory to merge any entities that are not part of the persistence context. Spring Batch provides a JpaItemWriterBuilder to construct an instance of the JpaItemWriter.

The LdifReader reads LDIF (LDAP Data Interchange Format) records from a Resource, parses them, and returns a LdapAttribute object for each read executed. Spring Batch provides a LdifReaderBuilder to construct an instance of the LdifReader.

The MappingLdifReader reads LDIF (LDAP Data Interchange Format) records from a Resource, parses them then maps each LDIF record to a POJO (Plain Old Java Object). Each read returns a POJO. Spring Batch provides a MappingLdifReaderBuilder to construct an instance of the MappingLdifReader.

The AvroItemReader reads serialized Avro data from a Resource. Each read returns an instance of the type specified by a Java class or Avro Schema. The reader may be optionally configured for input that embeds an Avro schema or not. Spring Batch provides an AvroItemReaderBuilder to construct an instance of the AvroItemReader.

The SimpleMailMessageItemWriter is an ItemWriter that can send mail messages. It delegates the actual sending of messages to an instance of MailSender. Spring Batch provides a SimpleMailMessageItemWriterBuilder to construct an instance of the SimpleMailMessageItemWriter.

The AvroItemWrite serializes Java objects to a WriteableResource according to the given type or Schema. The writer may be optionally configured to embed an Avro schema in the output or not. Spring Batch provides an AvroItemWriterBuilder to construct an instance of the AvroItemWriter.

The ScriptItemProcessor is an ItemProcessor that passes the current item to process to the provided script and the result of the script is returned by the processor. Spring Batch provides a ScriptItemProcessorBuilder to construct an instance of the ScriptItemProcessor.