This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0!

JSON and Text Document Readers

JSON

The JsonReader processes JSON documents, converting them into a list of Document objects.

Example

@Component
class MyJsonReader {

	private final Resource resource;

    MyJsonReader(@Value("classpath:bikes.json") Resource resource) {
        this.resource = resource;
    }

	List<Document> loadJsonAsDocuments() {
        JsonReader jsonReader = new JsonReader(this.resource, "description", "content");
        return jsonReader.get();
	}
}

Constructor Options

The JsonReader provides several constructor options:

  1. JsonReader(Resource resource)

  2. JsonReader(Resource resource, String…​ jsonKeysToUse)

  3. JsonReader(Resource resource, JsonMetadataGenerator jsonMetadataGenerator, String…​ jsonKeysToUse)

Parameters

  • resource: A Spring Resource object pointing to the JSON file.

  • jsonKeysToUse: An array of keys from the JSON that should be used as the text content in the resulting Document objects.

  • jsonMetadataGenerator: An optional JsonMetadataGenerator to create metadata for each Document.

Behavior

The JsonReader processes JSON content as follows:

  • It can handle both JSON arrays and single JSON objects.

  • For each JSON object (either in an array or a single object):

    • For each key in jsonKeysToUse that the object contains, it adds a key: value line to the content.

    • If no keys are specified, or the object contains none of them, the content is the whole object rendered as a Java Map string, such as {id=1, brand=Trek, description=…​}.

    • It generates metadata using the provided JsonMetadataGenerator (or an empty one if not provided).

    • It creates a Document object with the extracted content and metadata.

Using JSON Pointers

The JsonReader now supports retrieving specific parts of a JSON document using JSON Pointers. This feature allows you to easily extract nested data from complex JSON structures.

The get(String pointer) method

public List<Document> get(String pointer)

This method allows you to use a JSON Pointer to retrieve a specific part of the JSON document.

Parameters
  • pointer: A JSON Pointer string (as defined in RFC 6901) to locate the desired element within the JSON structure.

Return Value
  • Returns a List<Document> containing the documents parsed from the JSON element located by the pointer.

Behavior
  • The method uses the provided JSON Pointer to navigate to a specific location in the JSON structure.

  • If the pointer is valid and points to an existing element:

    • For a JSON object: it returns a list with a single Document.

    • For a JSON array: it returns a list of Documents, one for each element in the array.

  • If the pointer is invalid or points to a non-existent element, it throws an IllegalArgumentException.

Example
JsonReader jsonReader = new JsonReader(resource, "description");
List<Document> documents = this.jsonReader.get("/store/books/0");

Example JSON Structure

[
  {
    "id": 1,
    "brand": "Trek",
    "description": "A high-performance mountain bike for trail riding."
  },
  {
    "id": 2,
    "brand": "Cannondale",
    "description": "An aerodynamic road bike for racing enthusiasts."
  }
]

In this example, if the JsonReader is configured with "description" as the jsonKeysToUse, it creates one Document per bike in the array. The content of the first one is description: A high-performance mountain bike for trail riding., followed by a line separator.

Notes

  • The JsonReader uses Jackson for JSON parsing.

  • It reads the whole JSON resource into memory before it creates the documents.

  • If multiple keys are specified in jsonKeysToUse, the content has one key: value line per key, in the order of jsonKeysToUse.

  • The reader is flexible and can be adapted to various JSON structures by customizing the jsonKeysToUse and JsonMetadataGenerator.

Text

The TextReader processes plain text documents, converting them into a list of Document objects.

Example

@Component
class MyTextReader {

    private final Resource resource;

    MyTextReader(@Value("classpath:text-source.txt") Resource resource) {
        this.resource = resource;
    }

	List<Document> loadText() {
		TextReader textReader = new TextReader(this.resource);
		textReader.getCustomMetadata().put("filename", "text-source.txt");

		return textReader.read();
    }
}

Constructor Options

The TextReader provides two constructor options:

  1. TextReader(String resourceUrl)

  2. TextReader(Resource resource)

Parameters

  • resourceUrl: A string representing the URL of the resource to be read.

  • resource: A Spring Resource object pointing to the text file.

Configuration

  • setCharset(Charset charset): Sets the character set used for reading the text file. Default is UTF-8.

  • getCustomMetadata(): Returns a mutable map where you can add custom metadata for the documents.

Behavior

The TextReader processes text content as follows:

  • It reads the entire content of the text file into a single Document object.

  • The content of the file becomes the content of the Document.

  • Metadata is automatically added to the Document:

    • charset: The character set used to read the file (default: "UTF-8").

    • source: The filename of the source text file.

  • Any custom metadata added via getCustomMetadata() is included in the Document.

Notes

  • The TextReader reads the entire file content into memory, so it may not be suitable for very large files.

  • If you need to split the text into smaller chunks, you can use a text splitter like TokenTextSplitter after reading the document:

List<Document> documents = textReader.get();
List<Document> splitDocuments = TokenTextSplitter.builder().build().apply(this.documents);
  • The reader uses Spring’s Resource abstraction, allowing it to read from various sources (classpath, file system, URL, etc.).

  • Custom metadata can be added to all documents created by the reader using the getCustomMetadata() method.