Package org.springframework.ai.transformer.splitter

Class TokenTextSplitter

java.lang.Object

org.springframework.ai.transformer.splitter.TextSplitter

org.springframework.ai.transformer.splitter.TokenTextSplitter

All Implemented Interfaces:

Function<List<Document>,List<Document>>, DocumentTransformer

public class TokenTextSplitter
extends TextSplitter

Author:

Raphael Yu, Christian Tzolov

Constructor Summary

Constructors

Constructor	Description
TokenTextSplitter()
TokenTextSplitter(boolean keepSeparator)
TokenTextSplitter(int defaultChunkSize, int minChunkSizeChars, int minChunkLengthToEmbed, int maxNumChunks, boolean keepSeparator)

Method Summary

Modifier and Type	Method	Description
List<String>	split(String text, int chunkSize)
protected List<String>	splitText(String text)

Methods inherited from class org.springframework.ai.transformer.splitter.TextSplitter

apply, isCopyContentFormatter, setCopyContentFormatter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Function

andThen, compose

Constructor Details

TokenTextSplitter

public TokenTextSplitter()

TokenTextSplitter

public TokenTextSplitter(boolean keepSeparator)

TokenTextSplitter

public TokenTextSplitter(int defaultChunkSize,
 int minChunkSizeChars,
 int minChunkLengthToEmbed,
 int maxNumChunks,
 boolean keepSeparator)

Method Details

splitText

protected List<String> splitText(String text)

Specified by:

splitText in class TextSplitter

split

public List<String> split(String text,
 int chunkSize)