Package org.springframework.ai.tokenizer

Interface TokenCountEstimator

All Known Implementing Classes:
JTokkitTokenCountEstimator
public interface TokenCountEstimator
Estimates the number of tokens in a given text or message.
Since:
1.0.0
Author:
Christian Tzolov

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    estimate(Iterable<MediaContent> messages)
    Estimates the number of tokens in the given messages.
    int
    estimate(String text)
    Estimates the number of tokens in the given text.
    int
    estimate(MediaContent content)
    Estimates the number of tokens in the given message.

  • Method Details

    • estimate

      int estimate(String text)
      Estimates the number of tokens in the given text.
      Parameters:
      text - the text to estimate the number of tokens for.
      Returns:
      the estimated number of tokens.

    • estimate

      int estimate(MediaContent content)
      Estimates the number of tokens in the given message.
      Parameters:
      content - the content (Message or Document) to estimate the number of tokens for.
      Returns:
      the estimated number of tokens.

    • estimate

      int estimate(Iterable<MediaContent> messages)
      Estimates the number of tokens in the given messages.
      Parameters:
      messages - the messages to estimate the number of tokens for.
      Returns:
      the estimated number of tokens.