01Deep dive
Meaning and practical implications
There is no universal conversion between words and tokens. Use the target model’s tokenizer or provider usage record; language, formatting, and model-specific chat templates affect counts.
Tokens are units produced by a model’s tokenizer. A token may represent part of a word, a whole word, punctuation, or another supported input unit.
A short definition. Follow the related guide for a fuller explanation.
There is no universal conversion between words and tokens. Use the target model’s tokenizer or provider usage record; language, formatting, and model-specific chat templates affect counts.
Paste a week of traffic. Get the number that books the audit.
See what you're overpaying →