# Class Tokenizer

java.lang.Object
com.codename1.ai.Tokenizer

public final class Tokenizer extends java.lang.Object

Rough best-effort token counting. Useful for the common case of "am I likely to exceed this model's context window?" without shipping the full BPE table (cl100k_base is ~1.7 MB which is substantial for a mobile binary).

The rule of thumb is 1 token ~= 4 characters of English text, which holds within ~10-15% for typical chat traffic. For non-Latin scripts the ratio is closer to 1:1, so we clamp the lower bound at the rough number of words. Apps that need exact accounting should fetch a usage value from the API response and adjust their budget.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static int`
  `estimate(String text)`
  Approximate token count for `text`.
  `static int`
  `estimateMessages(List<ChatMessage> messages)`
  Estimate the prompt-tokens cost of an entire conversation.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Method Details

  - ### estimate

    public static int estimate(String text)
    Approximate token count for `text`.

  - ### estimateMessages

    public static int estimateMessages(List<ChatMessage> messages)
    Estimate the prompt-tokens cost of an entire conversation. Adds a small fixed overhead per message to approximate the role / formatting tokens the provider includes.