|
This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0! |
JSON and Text Document Readers
JSON
The JsonReader processes JSON documents, converting them into a list of Document objects.
Example
@Component
class MyJsonReader {
private final Resource resource;
MyJsonReader(@Value("classpath:bikes.json") Resource resource) {
this.resource = resource;
}
List<Document> loadJsonAsDocuments() {
JsonReader jsonReader = new JsonReader(this.resource, "description", "content");
return jsonReader.get();
}
}
Constructor Options
The JsonReader provides several constructor options:
-
JsonReader(Resource resource) -
JsonReader(Resource resource, String… jsonKeysToUse) -
JsonReader(Resource resource, JsonMetadataGenerator jsonMetadataGenerator, String… jsonKeysToUse)
Parameters
-
resource: A SpringResourceobject pointing to the JSON file. -
jsonKeysToUse: An array of keys from the JSON that should be used as the text content in the resultingDocumentobjects. -
jsonMetadataGenerator: An optionalJsonMetadataGeneratorto create metadata for eachDocument.
Behavior
The JsonReader processes JSON content as follows:
-
It can handle both JSON arrays and single JSON objects.
-
For each JSON object (either in an array or a single object):
-
For each key in
jsonKeysToUsethat the object contains, it adds akey: valueline to the content. -
If no keys are specified, or the object contains none of them, the content is the whole object rendered as a Java
Mapstring, such as{id=1, brand=Trek, description=…}. -
It generates metadata using the provided
JsonMetadataGenerator(or an empty one if not provided). -
It creates a
Documentobject with the extracted content and metadata.
-
Using JSON Pointers
The JsonReader now supports retrieving specific parts of a JSON document using JSON Pointers.
This feature allows you to easily extract nested data from complex JSON structures.
The get(String pointer) method
public List<Document> get(String pointer)
This method allows you to use a JSON Pointer to retrieve a specific part of the JSON document.
Parameters
-
pointer: A JSON Pointer string (as defined in RFC 6901) to locate the desired element within the JSON structure.
Return Value
-
Returns a
List<Document>containing the documents parsed from the JSON element located by the pointer.
Behavior
-
The method uses the provided JSON Pointer to navigate to a specific location in the JSON structure.
-
If the pointer is valid and points to an existing element:
-
For a JSON object: it returns a list with a single Document.
-
For a JSON array: it returns a list of Documents, one for each element in the array.
-
-
If the pointer is invalid or points to a non-existent element, it throws an
IllegalArgumentException.
Example JSON Structure
[
{
"id": 1,
"brand": "Trek",
"description": "A high-performance mountain bike for trail riding."
},
{
"id": 2,
"brand": "Cannondale",
"description": "An aerodynamic road bike for racing enthusiasts."
}
]
In this example, if the JsonReader is configured with "description" as the jsonKeysToUse, it creates one Document per bike in the array.
The content of the first one is description: A high-performance mountain bike for trail riding., followed by a line separator.
Notes
-
The
JsonReaderuses Jackson for JSON parsing. -
It reads the whole JSON resource into memory before it creates the documents.
-
If multiple keys are specified in
jsonKeysToUse, the content has onekey: valueline per key, in the order ofjsonKeysToUse. -
The reader is flexible and can be adapted to various JSON structures by customizing the
jsonKeysToUseandJsonMetadataGenerator.
Text
The TextReader processes plain text documents, converting them into a list of Document objects.
Example
@Component
class MyTextReader {
private final Resource resource;
MyTextReader(@Value("classpath:text-source.txt") Resource resource) {
this.resource = resource;
}
List<Document> loadText() {
TextReader textReader = new TextReader(this.resource);
textReader.getCustomMetadata().put("filename", "text-source.txt");
return textReader.read();
}
}
Constructor Options
The TextReader provides two constructor options:
-
TextReader(String resourceUrl) -
TextReader(Resource resource)
Parameters
-
resourceUrl: A string representing the URL of the resource to be read. -
resource: A SpringResourceobject pointing to the text file.
Configuration
-
setCharset(Charset charset): Sets the character set used for reading the text file. Default is UTF-8. -
getCustomMetadata(): Returns a mutable map where you can add custom metadata for the documents.
Behavior
The TextReader processes text content as follows:
-
It reads the entire content of the text file into a single
Documentobject. -
The content of the file becomes the content of the
Document. -
Metadata is automatically added to the
Document:-
charset: The character set used to read the file (default: "UTF-8"). -
source: The filename of the source text file.
-
-
Any custom metadata added via
getCustomMetadata()is included in theDocument.
Notes
-
The
TextReaderreads the entire file content into memory, so it may not be suitable for very large files. -
If you need to split the text into smaller chunks, you can use a text splitter like
TokenTextSplitterafter reading the document:
List<Document> documents = textReader.get();
List<Document> splitDocuments = TokenTextSplitter.builder().build().apply(this.documents);
-
The reader uses Spring’s
Resourceabstraction, allowing it to read from various sources (classpath, file system, URL, etc.). -
Custom metadata can be added to all documents created by the reader using the
getCustomMetadata()method.