What's a Token, and Why AI Has Limits

Part 2 of How AI Works. In part 1 we saw that a chatbot like ChatGPT, Claude or Gemini is a language model: a very good autocomplete. It reads what you wrote, gives every possible next piece a probability, picks one, adds it to the text and does it again. One piece at a time, until the answer is finished. This part is about those pieces. They’re called tokens, and they explain a lot of the strange things AI does.

In 2024, a simple question went around the internet: how many R’s are in the word “strawberry”?

The answer is three. For a long time, popular chatbots said two. In August 2024, TechCrunch ran a whole article about it, pointing out that models like GPT-4o and Claude got it wrong.

How can a system that writes essays and passes exams fail a question a seven-year-old gets right? It isn’t a bug someone forgot to fix. The model never saw the letters. It doesn’t read letters, and it doesn’t read words either. It reads tokens.

Once you understand tokens, three things make sense: why AI stumbles on spelling and counting, why some languages cost more than others, and why every AI tool has limits measured in tokens.

Why not letters? Why not words?

Before a language model can do anything with your text, it has to turn the text into numbers. A model is a large pile of maths: it can only multiply and add numbers. So the first question is how to chop text into pieces, so each piece can get a number.

There are two obvious options, and both have problems.

Option 1: one piece per letter. Every letter, space and comma gets its own number. The list of pieces is short, but your text becomes very long. Our running example for this series, “The cat sat on the mat because it was tired.”, would be 44 pieces instead of about 11. The model does work for every piece, so four times the pieces means far more work. It also has to learn from scratch that “c”, “a” and “t” together mean an animal.

Option 2: one piece per word. Now the sentence is short. But the list of words is endless. Names, typos, product codes, new slang, “ChatGPT”, Vietnamese place names, words in a hundred languages. Whatever isn’t on the list becomes an unknown blob, and the model can’t read it at all.

Tokens are the middle road. Common words get their own piece. Rare words get built from smaller, reusable pieces. And as a last resort, anything can be spelled out from very small pieces, so nothing is ever unreadable.

Picture a giant box of Lego bricks: big standard bricks for the shapes you build all the time, smaller ones for the odd corners. Every sentence, in any language, gets built from the same box.

How the pieces get chosen: byte-pair encoding

The most common way to build that box of bricks is called byte-pair encoding, or BPE. The idea is simple enough to do by hand.

1. Start with the smallest pieces: single characters (in modern versions, single bytes, the raw units computers store text in).
2. Look through a huge pile of text and find the pair of pieces that sits side by side most often. In English, that might be “t” followed by “h”.
3. Glue that pair into one new piece, “th”, and add it to the box.
4. Repeat. Soon “th” + “e” becomes “the”. Then “ the”, with a space in front, because “the” usually comes after a space.
5. Stop when the box has as many pieces as you want.

The original trick is a 1994 data compression method by Philip Gage. In 2016, Rico Sennrich and colleagues used it to split rare words into pieces for machine translation. OpenAI’s GPT-2 in 2019 ran it on raw bytes, which means any text at all, in any language or with any emoji, can be broken down without leftovers. GPT-2 ended up with 50,257 pieces. The tokenizer behind GPT-4o and several later OpenAI models, called o200k_base, has about 200,000.

The program that does the chopping is called a tokenizer. The pieces are tokens. And the box of all possible pieces is the vocabulary.

Two things are worth knowing about this:

• The tokenizer is built before the model learns anything. It’s chosen once, by counting which pieces are common in its training text, and then it stays fixed.
• Every company has its own. OpenAI, Anthropic and Google each use their own tokenizers, so the same sentence becomes a different number of tokens in different models. The idea is the same everywhere. The exact pieces are not.

From pieces to token IDs

Every piece in the vocabulary has a fixed number, like a page number in a dictionary. That number is the token ID. The model never sees your words. It sees a list of token IDs.

Here’s our sentence run through OpenAI’s open-source tokenizer, tiktoken, with the o200k_base encoding. These are the real pieces and real IDs, not an illustration:

A few things stand out.

The space belongs to the word after it. The piece isn’t “cat”, it’s “ cat” with the space in front. That’s why the diagram draws the space as a dot.

“the” and “The” are different tokens. “ the” in the middle of the sentence is 290. “The” at the start is 976. To the model, they’re two different entries. It learns that they mean the same thing from seeing both used, not because it can see they share letters.

This sentence is unusually tidy. It’s made of short, common English words, so every word is one token. Real text is messier, as we’ll see in a moment.

The IDs themselves mean nothing. 290 isn’t “smaller” than 25920 in any useful way. They’re just labels, like seat numbers. Turning those labels into something with meaning is the job of embeddings, the subject of part 3.

Where the Lego analogy breaks: Lego bricks are physical things you can look at. A token ID is just a number. The model can’t open token 9059 and see the letters c, a, t inside. Remember that, because it’s the whole reason for the next section.

Why “strawberry” broke it

Here’s what the tokenizer does with the strawberry question, measured the same way:

The video version of this post, and many explainers online, say GPT-4 saw “strawberry” as str, aw, berry. That’s true for the word typed on its own. But inside a normal question, with a space in front, both GPT-4’s and GPT-4o’s tokenizers turn “ strawberry” into one single token. The model gets one number, 101830 in the newer tokenizer, and is asked to count the R’s inside it.

Think of it like this. Someone shows you a sticker with a picture of a strawberry, and asks how many R’s are in it. You can’t count them from the sticker. You’d have to remember how the word is spelled, then count. The model is in the same position. It has to recall the spelling from patterns it saw during training, then count, and either step can slip. Two R’s sit side by side in “berry”, which makes them easy to lose.

The same blind spot shows up in other tasks:

• counting letters or words exactly,
• spelling a word backwards,
• writing “exactly 10 words” or “under 160 characters”,
• finding the fifth letter of a name,
• copying long numbers, codes and references, which get chopped into chunks like “123 · 456 · 7”.

Newer models often get strawberry right now, usually by writing the word out letter by letter before counting. That’s the trick you can use too: ask it to spell the word out first, or to use its code or calculator tool when it has one. And for anything that matters in your business, like phone numbers, booking references, invoice totals and character limits, check the result yourself. I wrote more about designing around this kind of weakness in treat the LLM like an unreliable API.

Not every language costs the same

The tokenizer learns its pieces from text, and the text it learns from has historically been dominated by English. So English words tend to get big, efficient pieces. Many other languages get chopped into smaller ones.

OpenAI’s own rule of thumb is that in English, one token is about four characters, and 100 tokens is about 75 words. Their help page also notes that other languages often need more tokens for the same text. A 2023 study from the University of Oxford (Petrov and colleagues, presented at NeurIPS) measured this across many languages and found that the same text can need very different numbers of tokens depending on the language, with differences of up to 15 times in some cases.

To see this for Vietnamese, here is one short guest welcome message in English, translated into Vietnamese and Thai, with the tokens counted with OpenAI’s two most recent tokenizers:

That’s one short message, so treat it as an example, not a study. But it shows two things clearly.

First, the gap is real. In the older tokenizer, Vietnamese words with tone marks were often broken into tiny bits: “mệt” (tired) became three pieces, “ m · ệ · t”. The Vietnamese message took almost twice as many tokens as the English one.

Second, it’s getting better. The newer tokenizer has a much bigger vocabulary, with room for many more non-English pieces, and the gap for this message shrank to about 10% for Vietnamese and about 30% for Thai. Other companies’ tokenizers will give different numbers again.

Why does it matter? More tokens for the same message means it costs more if you pay per token, fills the model’s memory faster and takes a little longer to generate. You can absolutely keep working in your own language. Just know that you’ll reach the limits a bit sooner.

The limits are all counted in tokens

Every limit you hit when using AI is measured in tokens. There are three.

1. How much it can read: the context window

Part 1 ended on this: the model only knows what’s in front of it, and there’s a limit to how much that can be. That limit is the context window, the model’s working memory. It’s the total amount of text the model can look at while writing its answer: the instructions the app adds, your messages, its own replies, any files you paste, and the answer it’s writing right now. Anthropic’s documentation defines it the same way, as everything the model can reference when generating a response, including the response itself.

Every model has a maximum, measured in tokens. At the time of writing (October 2026), Anthropic’s documentation lists several Claude models with a 1 million token window. Window sizes change often, so check your tool’s documentation for the model you actually use.

What happens when it fills up depends on where you are:

• Developers using the API get an error if the input alone is too long. If there’s room for the question but not the whole answer, the answer gets cut off when the window runs out.
• Chat apps like ChatGPT and Claude manage it for you. They might drop the oldest messages, keeping the most recent ones (Anthropic’s docs describe this as a “first in, first out” approach in chat interfaces), or summarise older parts of the conversation to save space. Either way, details from early in a long chat can quietly disappear.

There’s also a catch before the window is full. In 2023, researchers led by Nelson Liu at Stanford found that models use information at the start and end of a long input better than information in the middle. They called it “lost in the middle”. Anthropic’s documentation also warns that as the token count grows, accuracy and recall get worse, and calls this “context rot”. A bigger window is not the same as perfect attention to every word in it.

2. How much it can write

Each answer also has a maximum length in tokens. Developers set it with a parameter; chat apps set it for you. Ask for a 40-page report in one message, and you may get a reply that stops mid-sentence, or one that has quietly squashed 40 pages into 6.

The fix is to ask in parts: the outline first, then section one, then section two. If it stops, type “continue”. Breaking it up also gives you a chance to correct it before it goes too far.

3. How much you can use

Free chat plans limit how much you can send in a few hours. Paid plans have limits too, just higher. When you hit one, some apps switch you to a smaller model until the limit resets.

For businesses that connect AI to their own software through an API, the meter is explicit. Pricing is per token, usually quoted per million tokens, and input tokens (what you send) and output tokens (what it writes) are priced separately. On Anthropic’s price list at the time of writing, output tokens cost several times more than input tokens. Rate limits are counted in tokens too: Anthropic’s limits include input tokens per minute and output tokens per minute.

Why charge by the token? Because tokens are what the model actually processes. The model does its work once for every token it reads, and once for every token it writes, one token at a time. Tokens are the closest thing there is to a unit of work, so they’re what gets measured and billed.

Here’s the part most people miss. A model doesn’t remember your chat between messages. Each time you send a message, the app sends the whole conversation so far back to the model, so it can see it. Message one sends one message. Message twenty sends all twenty. A long chat gets more expensive with every turn, and fills the window faster. (Providers offer caching that makes repeated parts cheaper, but those tokens still take up space in the window.) I covered the money side of this in the hidden costs of AI tools.

Five token-smart habits

You don’t need to count tokens to work well with them. These five habits cover most of it.

1. Start a new chat for each new task. A fresh context window means fewer tokens, lower cost and sharper answers.
2. Paste only the part that matters. Ten relevant lines beat fifty pages, and they avoid “lost in the middle”.
3. Put your key instruction at the start, and repeat it at the end of long requests. (The 5-part prompt recipe helps here.)
4. Ask for long things in parts, and ask for short answers when short is all you need.
5. For counting, spelling and maths, ask it to go step by step or use a tool, and check anything that matters yourself.

Here’s how that plays out. Mai, who runs a hair salon (she’s a made-up example), gets a 20-page contract from a new product supplier. Her first try: paste all 20 pages and ask “anything I should worry about?” The answer is vague, and it misses a cancellation fee on page 14.

Her second try: a new chat, one section at a time, with her question at the top. “In this section, list any fees or penalties.” This time it finds the fee straight away, plus two more she hadn’t noticed. Then she checks each one against the contract itself, because the contract is what she signs, not the summary.

Same AI, same contract. She just worked with the tokens instead of against them.

The takeaway

AI doesn’t read letters or words. It reads tokens: chunks of text chosen by a tokenizer, each with a token ID, a plain number. That one fact explains a lot.

• It’s why AI can write a poem but struggles to count the R’s in strawberry.
• It’s why the same message can cost more in Vietnamese or Thai than in English.
• It’s why every limit, from the context window to the answer length to your bill, is counted in tokens.

Work in small, clear pieces, and most of those limits stop getting in your way.

But a list of numbers like 976, 9059, 10139 still doesn’t mean anything. How does the model know that “cat” and “kitten” are close, and “cat” and “invoice” are not? That’s the next step of our sentence’s journey.

Next: How AI Turns Words Into Numbers: Embeddings Explained.

Sources: OpenAI Help Center, “What are tokens and how to count them” (rules of thumb for English and the note on other languages), checked October 2026; OpenAI tiktoken library (github.com/openai/tiktoken), version 0.14.0, used to measure every token split, ID and count in this post with the o200k_base and cl100k_base encodings, October 2026; Philip Gage, “A New Algorithm for Data Compression,” The C Users Journal, 1994; Sennrich, Haddow and Birch, “Neural Machine Translation of Rare Words with Subword Units,” ACL 2016 (arXiv:1508.07909); Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2), OpenAI, 2019; Petrov, La Malfa, Torr and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023 (arXiv:2305.15425); Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” TACL 2024 (arXiv:2307.03172); Anthropic documentation, “Context windows,” “Pricing” and “Rate limits” (platform.claude.com/docs), checked October 2026; TechCrunch, “Why AI can’t spell ‘strawberry’,” 27 August 2024.

Related: How ChatGPT Actually Works, Explained Simply, Generative AI Large Language Model: Train From Pre-trained Model, and A Chatbot Sold a Car for $1: 5 AI Fails Every Business Should Know.