What Is a Token in AI?

A token is the unit an AI model actually reads — not a word, not a letter. It explains your bill, your context limit, and why models miscount letters.

You have probably met tokens three times without being introduced.

You saw pricing quoted “per million tokens.” You saw a spec sheet advertising a “200K token context window.” And someone told you a model miscounts letters “because of tokenization.” Three unrelated-looking conversations, one underlying unit.

A token is the unit of text a language model actually reads. Once you can see tokens, all three become the same conversation.

A token is a chunk, not a word

A model never receives your sentence as letters. Before anything else happens, your text is cut into pieces drawn from a fixed list called a vocabulary, and each piece is swapped for its ID number. That list is built once, when the model is created, and never changes afterward.

Those pieces are tokens. A token can be:

• a whole common word — water
• a word with its leading space — ␣water, typically a different token from water
• a fragment of a rare or long word — un + fathom + able
• a suffix — ing, ed, 's
• a punctuation mark, a digit, a line break

Two consequences follow immediately, and both surprise people.

Common words are cheap; rare words are expensive. cat is likely one token. An unusual surname or a technical term may be four or five — not because it is longer in letters, but because the vocabulary never earned a single entry for it.

Spaces travel with the token. Most modern tokenizers attach the leading space, which is why the same word can have different IDs depending on where it sits in a sentence.

Why not just use words? Or letters?

Both extremes were tried, and both hurt.

One token per word sounds natural but collapses on contact with reality. You would need an entry for every word, inflection, name, and typo. The list explodes, and anything missing becomes genuinely unrepresentable — the out-of-vocabulary problem.

One token per character never runs out of coverage, but makes every sequence far longer. Since attention cost grows faster than linearly with sequence length, this is expensive exactly where you cannot afford it.

Subword tokenization is the compromise that won: keep frequent words whole, break rare words into reusable fragments. The vocabulary stays a fixed, manageable size, and nothing is ever unrepresentable, because the worst case falls back to very small pieces.

Where the vocabulary comes from

The dominant method is Byte Pair Encoding, and the idea is short enough to state completely:

1. Start with the smallest units — bytes or characters.
2. Scan a large body of text. Find the most frequent adjacent pair.
3. Merge that pair into one new unit. Add it to the vocabulary.
4. Repeat until the vocabulary hits its target size.

Run this long enough and common words get merged into single units, while rare words remain assembled from fragments. Nobody hand-wrote the list; it fell out of the statistics of the training text.

That last point is worth carrying: the vocabulary is a fossil of the text it was built from. Whatever was common in that text is cheap forever after. Where Does the Vocabulary Come From? walks through this in detail.

Where tokens show up in your life

Your bill

APIs bill per token, not per word or per request. Input and output tokens are priced separately, with output consistently the more expensive of the two.

The practical consequence catches people out: a long document you send once can cost less than a short back-and-forth, because in a conversation the entire history is re-sent as input on every single turn. Cost grows with the square of conversation length, not linearly.

Your context window

A “200K context window” means 200K tokens, and that budget covers everything at once — system prompt, your question, the conversation so far, any pasted documents, plus room for the answer being generated. See What Is a Context Window?.

Failures that look like stupidity

Ask a model how many times r appears in strawberry and it may get it wrong. Ask it to reverse a word and it may stumble.

Part of the reason is that the model never received individual letters. It received a handful of IDs standing for chunks. Counting letters means reporting on something below the resolution of its own input.

This is worth stating carefully, because it is routinely overstated: tokenization is a real contributor, not the entire explanation. Models often can spell words correctly when asked directly, which means letter-level information is partially recoverable from the training data. The honest version is that tokenization makes character-level tasks unnaturally hard, not strictly impossible.

Tokens are not equally cheap in every language

Because the vocabulary is a fossil of its training text, and that text skewed heavily toward English, English earns the most whole-word entries. The same meaning expressed in another language — especially in a non-Latin script — generally costs more tokens.

You pay for that difference twice: once in money, once in context window. The size of the gap depends entirely on which tokenizer you are using, so measure it rather than trusting a quoted multiplier.

Try it yourself

Ten minutes with a real tokenizer beats another article. Most model providers publish a browser-based tokenizer; open one and:

1. Tokenize a sentence you wrote. Count the tokens against the words.
2. Tokenize a word with and without a leading space. Compare the IDs.
3. Tokenize a technical term from your own field. Watch it shatter.
4. Tokenize the same paragraph in two languages. Compare totals.

What to remember

• A token is a chunk of text from a fixed vocabulary — usually a common word, a fragment, or a word with its leading space.
• The vocabulary was frozen when the model was built and reflects the text it was built from, which is why common English is cheap.
• Cost, context limits, and a family of “why is it bad at this” failures are all denominated in this one unit.

Next: What Is an LLM? — what the machine reading these tokens actually is.