AI glossary Models & Architecture
What is a token in AI and LLMs?
← All glossary termsThe basic unit of text a language model reads and writes: a word, part of a word or a symbol. In English, one token is roughly four characters.
Explained
A token is the unit of text a language model actually processes. Before the model sees any text, a tokeniser splits it into pieces that can be whole words, parts of words, punctuation or spaces, and maps each piece to a number. In English, a common rule of thumb is about four characters, or three quarters of a word, per token; other languages and code often need more tokens for the same content. Token counts matter in practice because providers charge per token and because the context window, the amount a model can handle at once, is measured in tokens.
How tokenisation works
Most modern language models use subword tokenisation. Frequent words get a single token, while rare words are split into smaller pieces that the model has seen before. Methods such as byte pair encoding build this vocabulary by repeatedly merging the most common pairs of characters or fragments in a training corpus.
The result is that any text can be represented, including new words, names and typos, with a vocabulary of fixed size. Each model family has its own tokeniser, so the same sentence can produce different token counts in different models.
Why tokens matter for cost and limits
- Cost: APIs usually charge separately for input and output tokens, so long prompts and long answers cost more.
- Context window: the maximum number of tokens a model can consider in one request, including the prompt, retrieved documents, history and the answer.
- Latency: generating more output tokens takes more time.
- Language: Spanish and other languages often use more tokens than English for the same meaning, which affects both cost and how much fits in the window.
Practical tips
- Measure with the tokeniser of the model you use rather than estimating from words.
- Keep prompts and retrieved context focused: extra tokens add cost and can dilute the model's attention.
- Set limits on output length where long answers are not needed.
- Monitor token use per feature or user to spot unexpected growth.
Frequently asked questions
How many words is 1,000 tokens?
In English, roughly 750 words, using the common rule of thumb of about three quarters of a word per token. The exact figure depends on the text, the language and the model's tokeniser.
Why do LLMs use tokens instead of words?
Because a fixed vocabulary of subword pieces can represent any text, including rare words, names and typos, without needing an entry for every possible word. It is a balance between vocabulary size and sequence length.
Are tokens the same in every model?
No. Each model family has its own tokeniser and vocabulary, so the same text can produce different token counts. Use the tokeniser of the specific model to measure accurately.
How are tokens related to cost?
Most AI providers price their APIs per token, often with different rates for input and output. Total cost depends on prompt length, retrieved context, conversation history and the length of the answers.
Do Spanish texts use more tokens than English?
Usually yes. Tokenisers are often trained on more English text, so other languages tend to be split into more pieces. The difference depends on the tokeniser and the text.
Decision guides that use this term
Further reading
Related terms
Sources
- OpenAI · What are tokens and how to count them?
- Sennrich, Haddow and Birch · Neural Machine Translation of Rare Words with Subword Units (arXiv)
- Hugging Face · Tokenization algorithms (Transformers documentation)
Updated: 2026-10-21