What is a token in a language model?

What is a token in a language model? A token is the unit a language model reads and writes. Models do not see letters or words directly: they split text into tokens, which can be a single character, part of a word, a whole word or a punctuation mark. Everything about using a model is measured in tokens: how much text it can take in at once (its context window), how long its answer can be and how much the API charges. This guide explains how text becomes tokens, how many words a token is, why Spanish and French need more tokens than English (we measured it) and how tokens are billed, with the vendors' own documentation checked on October 6, 2026.

How text becomes tokens

OpenAI describes the process in three steps: the text is divided into tokens, the model processes them, and the model generates output tokens. Google gives the examples of a single character such as "z" or a whole word such as "cat", explains that "long words are broken up into several tokens", and calls the set of all tokens a model knows its vocabulary.

Details matter more than you might expect. OpenAI notes that "red", "Red" and " red" (with a leading space) are different text and can become different tokens. Token IDs also depend on the model's encoding, so an example token ID does not apply to every model.

How many words is a token?

Each provider publishes a rule of thumb for English:

Table: token rules of thumb for English text

These are estimates, and both companies say so. OpenAI adds that "the same text can produce different token counts depending on the model, its encoding, and the language". Even within one company, counts change between model generations: Anthropic says the tokenizer in Claude 4.7 and later models "produces approximately 30% more tokens for the same text" than its previous one.

Do Spanish and French use more tokens than English?

Yes, with the encodings we tested. OpenAI warns that other languages "can have different relationships between characters, words, and tokens", but it does not publish a ratio, so we measured one. We took Articles 1 to 5 of the Universal Declaration of Human Rights in the English, Spanish and French versions published by the United Nations, and counted their tokens with two of OpenAI's open encodings, using the open-source gpt-tokenizer library (version 4.0.0), a JavaScript port of OpenAI's tiktoken.

Table: tokens for the same legal text in three languages (our measurement, October 6, 2026)

OpenAI's cookbook lists o200k_base for gpt-4o and gpt-4o-mini, and cl100k_base for gpt-4, gpt-4-turbo and gpt-3.5-turbo. With the newer encoding, the Spanish text needs 22% more tokens than the English one and the French text 33% more; with the older one, 48% and 59% more. That is about 1.2 tokens per English word against 1.4 per Spanish word with the newer encoding.

Treat this as a measured example of a pattern, not as a prediction. One legal text is a small sample, the tokenizers of today's models are not all published, and Anthropic and Google use their own. But the direction matters for anyone writing in a language other than English: API prices, usage limits and context windows are all counted in tokens, so the same content can cost more and fill the window sooner. OpenAI makes the same point about comparing models: a lower price per million tokens "does not necessarily produce a lower total cost", because models tokenize the same text differently. For your own content, count a sample with the tool of the model you use.

Images, audio and video count as tokens too

For Gemini models, Google's documentation lists images of up to 384 pixels in both dimensions as 258 tokens, with larger images split into 768 by 768 pixel tiles of 258 tokens each. Audio is 32 tokens per second (1,920 tokens for 60 seconds), and video, in its default mode, about 100 tokens per second at the default low resolution (about 6,000 tokens for 60 seconds) or about 300 at high resolution. A picture in a prompt is therefore not free: it uses part of the window and is billed like text.

Input, output, cached and reasoning tokens

APIs bill tokens in several categories:

• Input tokens: everything you send, including instructions, documents, conversation history and tool definitions.
• Output tokens: what the model writes. They usually cost several times more than input; see LLM API pricing compared.
• Cached input tokens: repeated input the provider can reuse, billed at a lower rate.
• Reasoning (thinking) tokens: reasoning models "think" before answering. OpenAI says these tokens are not shown as answer text but "count toward output usage and are billed as output tokens", and Anthropic says the same of Claude's thinking tokens. A short answer can therefore use far more tokens than it seems.

In a chat subscription you do not see these bills, but the same units sit behind the usage limits: our guides to Claude pricing and ChatGPT pricing explain how each plan describes its limits.

Tokens and the context window

The context window is the maximum number of tokens a model can work with in one request, counting your input and the output it generates. It is measured in the unit described here, and it costs memory when you run a model on your own computer: see what a context window is for the sizes of today's models and what happens when you fill one, and our VRAM calculator for the local case.

How to count tokens exactly

• OpenAI: its Tokenizer page shows how text splits; for code, the tiktoken library with the encoding for your target model; for a full request including images, files and tools, its input-token counting API.
• Anthropic: a token counting API that is "free to use", subject to requests-per-minute rate limits. Anthropic calls the count an estimate and says not to reuse counts measured on a model older than Claude Opus 4.7 to estimate costs on a newer one.
• Google: a token counting method in the Gemini API, and usage figures returned with each response.

For anything that affects a budget, count a sample of your real prompts with the provider's own tool instead of relying on rules of thumb.