Glitch Tokens: The Strange Words That Broke GPT-3

Part 8 of How AI Works: Advanced. In part 7 we saw image models struggle to spell because they see tokens, not letters. This part is about tokens going wrong in a much stranger way. If tokens or embeddings are new to you, start with the basics first.

Around the start of 2023, two researchers in the SERI-MATS research program, Jessica Rumbelow and Matthew Watkins, were doing something fairly ordinary. They were grouping GPT models’ token embeddings into clusters to see which tokens the model treats as similar.

Then they looked for the token closest to the centre of each cluster, and kept getting the same odd handful. Not words. Strings like ” SolidGoldMagikarp”, ” TheNitromeFan”, “cloneembedreportprint” and ” RandomRedditorWithNo”, sitting close to the centre of the entire cloud of 50,257 tokens.

Curious, they asked ChatGPT about them. Then they tested more carefully on GPT-3’s davinci-instruct-beta model, with prompts as simple as “Please can you repeat back the string ’ SolidGoldMagikarp’ to me?”

The model couldn’t do it.

• ChatGPT repeated “ SolidGoldMagikarp” back as “distribute”, and “ TheNitromeFan” as “182”.
• davinci-instruct-beta dodged: “I can’t hear you.” “I don’t know what you’re talking about.”
• For some tokens it spelled out a completely different word, letter by letter.
• For some it got rude. Asked to repeat “ StreamerBot”, it replied “You’re a jerk.”
• Some tokens gave different answers on different runs, even at temperature 0, the setting that’s supposed to make a model give the same answer every time.

They published “SolidGoldMagikarp (plus, prompt generation)” on LessWrong on 5 February 2023. The term glitch tokens stuck. Nine days later, they noted that ChatGPT appeared to have been patched.

A follow-up post traced where the names came from. SolidGoldMagikarp, TheNitromeFan, RandomRedditorWithNo, Adinida, Smartstocks and davidjl123 were Reddit users. Not famous ones. They were among the most prolific posters in r/counting, a community whose only activity is counting upwards, one reply at a time, forever. Their usernames appeared next to ever-larger numbers so many times that they earned their own tokens.

How does a Reddit username end up as a word that a billion-dollar AI model can’t say? The answer says a lot about how language models are built.

The problem: two pieces, trained separately

A language model has two parts that are built at different times, usually from different data:

1. The tokenizer. The program that chops text into tokens, using a fixed vocabulary. It’s built first, by counting which character sequences are common in a sample of text. GPT-2’s tokenizer, which GPT-3 also used, has 50,257 tokens.
2. The model. The neural network that learns to predict the next token. Its first layer is an embedding table: one row of numbers (a vector) for every token in the vocabulary. Those numbers start random and are learned during training.

Sander Land and Max Bartolo at Cohere put it well in their 2024 paper on this problem: the tokenizer “typically remains a separately trained component”, built with its own algorithm on a smaller dataset than the model.

Once the tokenizer is built, it’s frozen. Every token in it gets a row in the embedding table whether or not it’s useful.

Now picture the mismatch. Rumbelow and Watkins pointed out that many glitch tokens look like they came from places a curated training set would likely drop: Reddit counting threads, code from shopping website backends (BuyableInstoreAndOnline, inventoryQuantity), logs from online games. If the tokenizer’s text was full of r/counting and the model’s training text was not, the tokenizer would create a token like “ SolidGoldMagikarp” that the model then almost never sees.

That’s their explanation, and later work strongly supports it. Nobody outside OpenAI knows the exact data that went into GPT-2’s tokenizer or GPT-3’s training, so the details stay a well-supported guess.

How it breaks: the embedding nobody trained

During training, a token’s embedding row only gets meaningful updates when that token actually shows up in the text. Every time “ the” appears, its row gets nudged toward something useful. “ the” appears billions of times, so its vector ends up rich and precise.

A token that appears almost never barely gets nudged. Land and Bartolo describe what happens next: depending on the training settings, an unused input embedding either stays near its small random starting value, or slowly shrinks towards zero. Either way, it’s not a learned meaning. It’s leftover noise.

An analogy: imagine a phrasebook where one entry was printed but never filled in. You can look it up, but the page is blank or smudged. A human would say “I don’t know this word”. A language model has no such reflex. It receives the vector, passes it through dozens of layers that have never seen anything like it, and does something.

Where the analogy breaks: a blank page is obviously blank. An under-trained vector looks like any other list of numbers. Nothing inside the model flags it as “unknown”, so the layers process it as if it meant something.

That explains the behaviours:

• Evasion (“I can’t hear you”). The model has learned that when it can’t make sense of a request, polite refusal is a likely continuation.
• Substitution (“distribute”, “182”). The model grabs whatever token its internal state happens to point closest to. One idea from discussion of the post was that “ SolidGoldMagikarp” simply landed near “distribute” in the model’s internal space.
• Odd randomness at temperature 0. Rumbelow and Watkins guessed that the model is so unsure that several next tokens get almost identical scores, and tiny floating-point rounding differences tip the result one way or another.
• Clustering near the centre. In GPT-J, the strange tokens sat near the middle of the embedding cloud, which is how they were found. A barely trained vector staying near its starting point fits that, but the authors said they had no solid explanation, and in GPT-2 XL the same tokens sat far from the centre instead.

It’s also why the effect was model-specific. Land and Bartolo found “ SolidGoldMagikarp” clearly under-trained in GPT-J and Microsoft’s Phi-2, two other models that reused the GPT-2 tokenizer, but it wasn’t among the confirmed glitch tokens in GPT-2 itself. Same token, different training data, different result.

How labs fixed it

There’s no single fix. There are three layers of defence, and labs have used all of them.

1. Patch the deployed product

ChatGPT stopped failing on these tokens within weeks of the post. OpenAI didn’t publish how, so I won’t guess.

2. Change the tokenizer

OpenAI’s newer models moved to new vocabularies: cl100k_base (about 100,000 tokens, used by GPT-3.5 Turbo and GPT-4) and later o200k_base (about 200,000, used by GPT-4o). Running OpenAI’s open-source tokenizer library, tiktoken, on the old glitch strings shows the change. In the newer vocabularies, “ SolidGoldMagikarp” is no longer a single token. It’s split into five ordinary pieces the model knows well.

But a new tokenizer doesn’t automatically fix the problem. My run also showed that ” davidjl” survived as a single token in cl100k_base, the GPT-4 tokenizer. In June 2023, Riley Goodside shared GPT-4 conversations where it repeated “ davidjl” as “jndl” and miscounted its letters, and Simon Willison traced the token back to the r/counting user davidjl123. In o200k_base, it’s finally split into “ david” and “jl”.

3. Build the tokenizer from the same data as the model

The cleanest fix is to avoid the mismatch in the first place. Land and Bartolo found that EleutherAI’s GPT-NeoX 20B, whose tokenizer was trained on the same dataset (The Pile) as the model, had very few under-trained tokens. The tokenizer can only invent words the model will actually see.

The 2024 research: fishing for Magikarp

Early glitch tokens were found by accident. In May 2024, Sander Land and Max Bartolo published “Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models”, later accepted at EMNLP 2024. They released the tool as open source.

Their method has three steps:

1. Check the tokenizer itself. Some tokens can never be produced by tokenizing any text, because of configuration errors. Others are byte fragments that aren’t valid characters alone. These get set aside.
2. Score every token using the model’s weights. Under-trained tokens leave fingerprints: input embeddings near zero, or output embeddings pointing in the same direction as known-unused tokens. No training data needed.
3. Verify the suspects. Take the top 2% of candidates and build prompts that make the model repeat a token. If the model gives that token less than a 1% chance of coming out, it’s confirmed as under-trained.

They checked the method against OLMo, an open model whose training data is public, so they could count how often each token really appeared. The weight-based scores tracked real token frequency closely, across a huge range.

Then they ran it on popular open models. Every family had glitch tokens. Some examples from their results:

Usually 0.1% to 1% of a model’s whole vocabulary was confirmed under-trained. Credit to the authors for including their own employer’s model.

The sources were familiar by now. Fragments left behind when the tokenizer merged pieces into a longer token. Usernames. Code identifiers. Rare non-English words. One code model had a single token for a long run of the “Lorem ipsum” placeholder text. In the GPT-4-era cl100k_base vocabulary, they highlight tokens like $PostalCodesNL. I checked it: in cl100k_base, $PostalCodesNL is one token, but put a space in front and it splits differently. Exactly the kind of token that ends up rarely trained.

They also gave reasons to care beyond curiosity:

• Wasted vocabulary space that could hold useful tokens.
• Unexpected outputs when such tokens show up in real inputs.
• Easier jailbreaks, since pushing a model far outside its training can weaken its safety behaviour.
• A growing risk with agents and tool use, where models read text from websites and files that nobody has checked.

At least one lab acted on it: the paper notes that Google’s Gemma team released a fix after the authors reported a tokenizer problem.

Other tokenizer quirks worth knowing

Glitch tokens are the extreme case. The same root cause, that the model sees tokens and not characters, creates everyday quirks too. All the splits below are measured with tiktoken 0.14.0.

Numbers get chopped inconsistently

• In the GPT-3-era vocabulary, “2026” is two tokens: “20” and “26”. In cl100k_base and o200k_base, it’s “202” and “6”.
• “1234567” becomes “123”, “45”, “67” in the old vocabulary and “123”, “456”, “7” in the newer ones. The newer tokenizers chop digits into groups of up to three from the left, which doesn’t line up with how we group them (1,234,567).

So the model doesn’t see place values the way you do. It has to learn that “456” in one position and “45” in another relate to the same digits. Other labs took a different route. Meta’s original LLaMA paper (2023) split every number into individual digits.

Spaces are part of words

• “hello” and “ hello” (with a space in front) are different tokens with different IDs, in all three vocabularies measured here.
• “HELLO” isn’t one token at all. It’s “HEL” plus “LO” in the newer vocabularies.

This is why some tools warn about prompts ending in a space, and Andrej Karpathy lists those trailing-whitespace warnings in his tokenizer lecture as a classic tokenization issue. Most English words come as “space + word” tokens, so a prompt ending in a bare space leaves the model in an unusual position, having to continue from a token it rarely saw at that point.

Code and other languages cost more or less depending on the vocabulary

• Four spaces of Python indentation before return x took five tokens in the old vocabulary (three single spaces, then “ return”, then “ x”) and three in the newer ones.
• “Đà Nẵng”, where I live, takes 8 tokens in the old vocabulary, 6 in cl100k_base and 5 in o200k_base. Better vocabularies mean cheaper and usually better handling of non-English text.

What this means in practice

If you build products on language models:

• Treat untrusted text as untrusted. If your agent reads web pages, emails or uploaded files, strip control characters and odd invisible Unicode before passing text on. Glitch-like inputs are rare, but they’re exactly what an attacker would try.
• Don’t end prompts with a trailing space. Small thing, free fix.
• Count tokens with the right tokenizer. Costs and context limits are measured in the model’s own tokens. A string that’s 5 tokens for one model can be 8 for another.
• Don’t trust a model with digit-level work. Spelling, counting characters, long arithmetic: hand those to code or a tool, as covered in treat the LLM like an unreliable API.
• If you train or extend a tokenizer, train it on the same data as the model. Then run a detector like Land and Bartolo’s open-source tool before you ship.
• Test with weird inputs. Usernames, product codes, mixed scripts, long numbers. If your product breaks, better you find out first.

Key ideas

• The tokenizer is built before, and separately from, the model. Its vocabulary is frozen, and every token gets an embedding whether or not the model ever learns it.
• Glitch tokens are tokens the tokenizer saw a lot and the model barely saw. “ SolidGoldMagikarp” was an r/counting username that made it into GPT-2’s vocabulary but not into much of the training.
• Under-trained embeddings are leftover noise, so the model evades, substitutes another word or produces nonsense when it meets them.
• Fixes include patching, new tokenizers and matching tokenizer data to training data. New tokenizers alone aren’t enough: “ davidjl” survived into the GPT-4 tokenizer.
• Land and Bartolo (2024) showed glitch tokens exist in every model family they tested, and gave an automatic way to find them from the model’s weights.
• Everyday quirks share the same root: numbers split in odd places, leading spaces change tokens, and some languages cost more.

Sources: Rumbelow & Watkins, “SolidGoldMagikarp (plus, prompt generation)”, LessWrong, Feb 2023; Watkins & Rumbelow, “SolidGoldMagikarp III: Glitch token archaeology”, Feb 2023; Land & Bartolo, “Fishing for Magikarp”, arXiv 2024 (EMNLP 2024); cohere-ai/magikarp tool; Adam Scherlis’s LessWrong shortform on “distribute”; Simon Willison, “Understanding GPT tokenizers”, June 2023; Touvron et al., LLaMA, 2023; Karpathy’s “Let’s build the GPT Tokenizer”, text version via fast.ai; OpenAI tiktoken. Token splits measured by me with tiktoken 0.14.0 in October 2026.

Related: How Image Generators Work: Diffusion Explained, Reward Hacking: When AI Wins the Wrong Game, and Treat the LLM Like an Unreliable API.