Package org.springframework.ai.tokenizer

Class JTokkitTokenCountEstimator

java.lang.Object

org.springframework.ai.tokenizer.JTokkitTokenCountEstimator

All Implemented Interfaces:

TokenCountEstimator

public class JTokkitTokenCountEstimator
extends Object
implements TokenCountEstimator

Estimates the number of tokens in a given text or message using the JTokkit encoding library.

Since:

1.0.0

Author:

Christian Tzolov, Soby Chacko

Constructor Summary

Constructors

Constructor	Description
JTokkitTokenCountEstimator()
JTokkitTokenCountEstimator(com.knuddels.jtokkit.api.EncodingType tokenEncodingType)

Method Summary

Modifier and Type	Method	Description
int	estimate(Iterable<MediaContent> contents)	Estimates the number of tokens in the given messages.
int	estimate(String text)	Estimates the number of tokens in the given text.
int	estimate(MediaContent content)	Estimates the number of tokens in the given message.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

JTokkitTokenCountEstimator

public JTokkitTokenCountEstimator()

JTokkitTokenCountEstimator

public JTokkitTokenCountEstimator(com.knuddels.jtokkit.api.EncodingType tokenEncodingType)

Method Details

estimate

public int estimate(String text)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given text.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

text - the text to estimate the number of tokens for.

Returns:

the estimated number of tokens.

estimate

public int estimate(MediaContent content)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given message.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

content - the content (Message or Document) to estimate the number of tokens for.

Returns:

the estimated number of tokens.

estimate

public int estimate(Iterable<MediaContent> contents)

Description copied from interface: TokenCountEstimator

Estimates the number of tokens in the given messages.

Specified by:

estimate in interface TokenCountEstimator

Parameters:

contents - the messages to estimate the number of tokens for.

Returns:

the estimated number of tokens.