|
This version is still in development and is not considered stable yet. For the latest stable version, please use Spring AI 2.1.0! |
HTML (JSoup)
The JsoupDocumentReader processes HTML documents, converting them into a list of Document objects using the JSoup library.
Dependencies
Add the dependency to your project using Maven or Gradle.
-
Maven
-
Gradle
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-jsoup-document-reader</artifactId>
</dependency>
dependencies {
implementation 'org.springframework.ai:spring-ai-jsoup-document-reader'
}
Example
@Component
class MyHtmlReader {
private final Resource resource;
MyHtmlReader(@Value("classpath:/my-page.html") Resource resource) {
this.resource = resource;
}
List<Document> loadHtml() {
JsoupDocumentReaderConfig config = JsoupDocumentReaderConfig.builder()
.selector("article p") // Extract paragraphs within <article> tags
.charset("ISO-8859-1") // Use ISO-8859-1 encoding
.includeLinkUrls(true) // Include link URLs in metadata
.metadataTags(List.of("author", "date")) // Extract author and date meta tags
.additionalMetadata("source", "my-page.html") // Add custom metadata
.build();
JsoupDocumentReader reader = new JsoupDocumentReader(this.resource, config);
return reader.get();
}
}
The JsoupDocumentReaderConfig allows you to customize the behavior of the JsoupDocumentReader:
-
charset: Specifies the character encoding of the HTML document (defaults to "UTF-8"). -
selector: A JSoup CSS selector to specify which elements to extract text from (defaults to "body"). -
separator: The string used to join text from multiple selected elements (defaults to "\n"). -
allElements: Iftrue, extracts all text from the<body>element, ignoring theselector(defaults tofalse). -
groupByElement: Iftrue, creates a separateDocumentfor each element matched by theselector(defaults tofalse). -
includeLinkUrls: Iftrue, extracts absolute link URLs and adds them to the metadata (defaults tofalse). -
metadataTags: A list of<meta>tag names to extract content from (defaults to["description", "keywords"]). -
additionalMetadata: Allows you to add custom metadata to all createdDocumentobjects.
Sample Document: my-page.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>My Web Page</title>
<meta name="description" content="A sample web page for Spring AI">
<meta name="keywords" content="spring, ai, html, example">
<meta name="author" content="John Doe">
<meta name="date" content="2024-01-15">
<link rel="stylesheet" href="style.css">
</head>
<body>
<header>
<h1>Welcome to My Page</h1>
</header>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="/about">About</a></li>
</ul>
</nav>
<article>
<h2>Main Content</h2>
<p>This is the main content of my web page.</p>
<p>It contains multiple paragraphs.</p>
<a href="https://www.example.com">External Link</a>
</article>
<footer>
<p>© 2024 John Doe</p>
</footer>
</body>
</html>
Behavior:
The JsoupDocumentReader processes the HTML content and creates Document objects based on the configuration:
-
The
selectordetermines which elements are used for text extraction. -
If
allElementsistrue, all text within the<body>is extracted into a singleDocument. -
If
groupByElementistrue, each element matching theselectorcreates a separateDocument. -
If neither
allElementsnorgroupByElementistrue, text from all elements matching theselectoris joined using theseparator. -
The document title, content from specified
<meta>tags, and (optionally) link URLs are added to theDocumentmetadata. -
The base URI, for resolving relative links, will be extracted from URL resources.
The reader preserves the text content of the selected elements, but removes any HTML tags within them.