Package org.springframework.ai.transformer.splitter

Class TokenTextSplitter

java.lang.Object

org.springframework.ai.transformer.splitter.TextSplitter

org.springframework.ai.transformer.splitter.TokenTextSplitter

All Implemented Interfaces:

Function<List<Document>,List<Document>>, DocumentTransformer

public class TokenTextSplitter
extends TextSplitter

A TextSplitter that splits text into chunks of a target size in tokens.

Author:

Raphael Yu, Christian Tzolov, Ricken Bazolo, Seunghwan Jung, Jemin Huh

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	TokenTextSplitter.Builder

Constructor Summary

Constructors

Constructor	Description
TokenTextSplitter()	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(boolean keepSeparator)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(int chunkSize, int minChunkSizeChars, int minChunkLengthToEmbed, int maxNumChunks, boolean keepSeparator, List<Character> punctuationMarks)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.
TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType, boolean keepSeparator)	Deprecated, for removal: This API element is subject to removal in a future version. since 2.0.0-M3, use builder() instead.

Method Summary

Modifier and Type	Method	Description
static TokenTextSplitter.Builder	builder()
protected List<String>	doSplit(String text, int chunkSize)	Splits text into chunks based on token count, using the overlap configured on this splitter.
protected List<String>	doSplit(String text, int chunkSize, int chunkOverlap)	Splits text into chunks based on token count.
protected int	getLastPunctuationIndex(String chunkText)
protected List<String>	splitText(String text)

Methods inherited from class org.springframework.ai.transformer.splitter.TextSplitter

apply, isCopyContentFormatter, setCopyContentFormatter, split, split

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.ai.document.DocumentTransformer

transform

Methods inherited from interface java.util.function.Function

andThen, compose

Constructor Details

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter()

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(boolean keepSeparator)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(com.knuddels.jtokkit.api.EncodingType encodingType,
 boolean keepSeparator)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

TokenTextSplitter

@Deprecated(since="2.0.0-M3",
            forRemoval=true)
public TokenTextSplitter(int chunkSize,
 int minChunkSizeChars,
 int minChunkLengthToEmbed,
 int maxNumChunks,
 boolean keepSeparator,
 List<Character> punctuationMarks)

Deprecated, for removal: This API element is subject to removal in a future version.

since 2.0.0-M3, use builder() instead.

Method Details

builder

public static TokenTextSplitter.Builder builder()

splitText

protected List<String> splitText(String text)

Specified by:

splitText in class TextSplitter

doSplit

protected List<String> doSplit(String text,
 int chunkSize)

Splits text into chunks based on token count, using the overlap configured on this splitter.

Subclasses that override this method replace the whole algorithm, so overlap does not apply to them unless they call doSplit(String, int, int).

Parameters:

text - the text to split

chunkSize - the target chunk size in tokens

Returns:

list of text chunks

doSplit

protected List<String> doSplit(String text,
 int chunkSize,
 int chunkOverlap)

Splits text into chunks based on token count.

Punctuation-based splitting only applies when the token count exceeds the chunk size (tokens.size() > chunkSize). Text that exactly matches or is smaller than the chunk size is returned as a single chunk without punctuation-based truncation.

Parameters:

text - the text to split

chunkSize - the target chunk size in tokens

chunkOverlap - the number of tokens each chunk shares with the one before it

Returns:

list of text chunks

Since:

2.1.0

getLastPunctuationIndex

protected int getLastPunctuationIndex(String chunkText)