Package org.springframework.ai.tokenizer

Class JTokkitTokenCountEstimator

java.lang.Object

org.springframework.ai.tokenizer.JTokkitTokenCountEstimator

All Implemented Interfaces:

TokenCountEstimator

public class JTokkitTokenCountEstimator
extends Object
implements TokenCountEstimator

Estimates the number of tokens in a given text or message using the JTokkit encoding library.

Since:

1.0.0

Author:

Christian Tzolov, Soby Chacko

Constructor Summary

Constructors

Constructor	Description
JTokkitTokenCountEstimator()	Creates a new JTokkitTokenCountEstimator with default CL100K_BASE encoding.
JTokkitTokenCountEstimator(com.knuddels.jtokkit.api.EncodingType tokenEncodingType)	Creates a new JTokkitTokenCountEstimator with the specified encoding type.

Method Summary

Modifier and Type	Method	Description
int	estimate(Iterable<MediaContent> contents)	Estimates the number of tokens in the given messages.
int	estimate(String text)	Estimates the number of tokens in the given text.
int	estimate(MediaContent content)	Estimates the number of tokens in the given message.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

JTokkitTokenCountEstimator

public JTokkitTokenCountEstimator()

Creates a new JTokkitTokenCountEstimator with default CL100K_BASE encoding.

JTokkitTokenCountEstimator

public JTokkitTokenCountEstimator(com.knuddels.jtokkit.api.EncodingType tokenEncodingType)

Creates a new JTokkitTokenCountEstimator with the specified encoding type.

Parameters:

tokenEncodingType - the encoding type to use for token counting

Method Details

estimate

public int estimate(String text)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given text.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

text - the text to estimate the number of tokens for.

Returns:

the estimated number of tokens.

estimate

public int estimate(MediaContent content)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given message.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

content - the content (Message or Document) to estimate the number of tokens for.

Returns:

the estimated number of tokens.

estimate

public int estimate(Iterable<MediaContent> contents)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given messages.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

contents - the messages to estimate the number of tokens for.

Returns:

the estimated number of tokens.