Synthetic Data and Model Collapse

The prompt was a dry paragraph about church towers.

It came from a Wikipedia-style article about medieval English architecture: towers begun before 1360, the Perpendicular style, master masons. A team of researchers fed it to a small language model and asked it to continue. Then they did something unusual. They trained a new model on text written by the first model. Then a third model on text from the second. And so on, generation after generation, each one learning only from its parent.

By the first generation the continuation had already wandered to St. Peter’s Basilica in Rome. By generation five it was talking about something being translated into more than 100 languages. By generation nine, the model’s answer to the same church-tower prompt was a list of rabbits: “black-tailed jackrabbits, white-tailed jackrabbits, blue-tailed jackrabbits, red-tailed jackrabbits”, and on into yellow.

That example is from a paper by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, published in Nature on 24 July 2024 under a blunt title: “AI models collapse when trained on recursively generated data.” An earlier version had circulated since May 2023 as “The Curse of Recursion.” The jackrabbits made headlines because they are funny. The real finding is quieter and more important.

If tokens, pre-training and fine-tuning are new to you, start with the basics first. This post assumes you know how a language model learns from text.

The problem: the internet is starting to eat its own cooking

Every large language model so far has been built mostly on human writing: books, websites, forums, code. That was the whole bet. Read enough of what people wrote, and you learn how people write and think.

But since late 2022, a growing share of new text online has been written by AI. Product descriptions, SEO articles, social posts, homework, the email you got this morning. Future models will be trained on a web that contains a lot of output from past models, whether anyone plans it or not.

So the question is simple and slightly scary: what happens when a model learns from another model’s output, over and over?

The Shumailov paper’s answer, in one line: it slowly forgets the rare stuff, and then it forgets almost everything.

What “model collapse” actually means

The authors define two stages.

Early collapse. The model starts losing information about the tails of the distribution. “Tails” means the rare cases: unusual words, uncommon facts, minority viewpoints, odd phrasings, the 1-in-10,000 example. The model still looks fine on average. You might not notice anything.
Late collapse. After enough generations, the model converges to something that looks little like the original data. Its outputs get narrower and more repetitive, with much less variety. In their words, it drifts toward a kind of single “point estimate,” one bland answer to everything.

That ordering matters. Collapse doesn’t start with gibberish. It starts with the edges quietly disappearing. By the time you get jackrabbits, the damage has been happening for a while.

They showed this in three kinds of models: simple statistical models (Gaussian mixtures), image models (variational autoencoders, which learn to compress and regenerate images), and language models. For the language model test they took OPT-125m, a small open model from Meta, fine-tuned it on a Wikipedia text dataset (wikitext2), and then trained each new generation on text produced by the previous one.

The results depended a lot on one setting:

No original data kept. Each generation trained only on the previous model’s output. Quality fell steadily. Perplexity, a standard measure of how surprised the model is by real text (lower is better), rose from about 20 to about 28 points over the generations.
10% of the original human data kept in every generation. The authors report only minor degradation.

Hold on to that second line. It is the most useful sentence in this post, and we will come back to it.

Model collapse, sketched: the rare cases vanish first, then the variety shrinks toward one typical answer.

Why it happens: the jar of marbles

Here is the simplest way I know to picture it.

Imagine a jar with 1,000 marbles. Most are red, blue and green. Ten are purple. Purple is your “tail,” the rare case.

Now play a game. Draw 100 marbles at random. Build a brand new jar of 1,000 marbles in exactly the same colour proportions as your 100. Throw the old jar away. Repeat.

With only ten purple marbles in a thousand, there’s a real chance (roughly one in three) that a draw of 100 contains no purple at all. When that happens, the new jar has zero purple. And here is the key point: once a colour hits zero, it can never come back. No future draw can produce a marble that isn’t in the jar.

Run the game long enough and the jar drifts toward fewer and fewer colours. Biologists know this as genetic drift in small populations. The same thing happens to a model trained on its own samples.

The paper names three sources of error that pile up generation after generation:

Sampling error. You only ever train on a finite sample, so rare things get under-counted or missed. This is the marble jar.
Expressivity error. The model can’t represent the real distribution perfectly. It has limited capacity, so it smooths things out.
Learning error. Training methods themselves (like stochastic gradient descent, the standard way models are trained) have their own biases and imperfections.

Any one of these is small. The problem is that each generation inherits the previous generation’s mistakes as if they were truth, then adds its own.

In practice there’s a fourth push that makes it worse. When people generate text with a model, they usually prefer the likely, polished, “safe” output. Lower sampling temperature and filters for quality both favour the middle of the distribution. So the text that ends up published online is already a bit narrower than what the model could produce.

Where the marble analogy breaks: a language model isn’t counting marbles. It generalises, so it can sometimes produce things it never saw exactly. And real training doesn’t throw away the old jar, which turns out to be the whole story.

The counter-evidence: just don’t throw away the real data

The Shumailov setup is a deliberately harsh scenario. Each generation trains only on the previous generation’s output, and the original human data is thrown away. That is a useful stress test. It is not how anyone actually builds models, and it is not how the web works. Old human text doesn’t disappear when new AI text appears.

In April 2024, a group led by Matthias Gerstgrasser and Rylan Schaeffer (with co-authors including David Donoho and Sanmi Koyejo) published “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.” They compared two setups:

Replace. Each generation’s synthetic data replaces the data before it. Result: collapse, matching the earlier work.
Accumulate. Each generation’s synthetic data is added to everything that came before, including the original real data. Result: no collapse. For a simple mathematical model they even proved that the test error has a fixed upper limit, no matter how many generations you run.

The same loop, two policies: replacing real data with synthetic data collapses; adding synthetic data to the real data doesn't.

This lines up with two other results:

Shumailov’s own paper: keeping just 10% of the original data per generation led to only minor degradation.
A July 2023 paper from Rice University, “Self-Consuming Generative Models Go MAD” (Alemohammad and colleagues), studied image models in a loop. Their conclusion: without enough fresh real data in each generation, future models lose either quality or diversity. They named it Model Autophagy Disorder, a nod to mad cow disease.

So the honest summary of the research is: collapse is real, but it’s a property of a closed loop. It happens when a model’s only food is its own output. Keep real data in the mix and the effect shrinks a lot. What the “accumulate” result doesn’t promise is that the tails are perfectly preserved forever. Rare cases can still get diluted when the pile grows mostly with synthetic text. That part is still being studied.

How labs use synthetic data successfully

Here is the twist. While researchers were warning about collapse, AI labs were using more synthetic data than ever, and getting good results. These aren’t contradictions. Look at how they do it.

1. Distillation: a strong teacher writes for a small student

Distillation means training a smaller model on the outputs of a bigger, stronger one. (Part 4 of this series covers it in depth.)

The clearest public example is DeepSeek-R1, published in January 2025. The team used R1 to generate about 800,000 training samples (roughly 600,000 reasoning examples and 200,000 general ones), then fine-tuned six smaller open models on them, based on Qwen2.5 and Llama, from 1.5 billion to 70 billion parameters. The small models gained strong reasoning skills from that synthetic data alone, with plain fine-tuning and no reinforcement learning step.

This isn’t a closed loop. The student learns from a better teacher. Information flows downhill from a stronger model, not in a circle.

2. Synthetic textbooks: designed, not scraped

In June 2023, Microsoft Research released “Textbooks Are All You Need,” introducing phi-1, a 1.3-billion-parameter coding model. Part of its training data was about 1 billion tokens of Python “textbooks” written by GPT-3.5, plus about 180 million tokens of synthetic exercises with solutions. The authors steered variety on purpose, for example by asking for different topics and target audiences.

phi-1 scored 50.6% on the HumanEval coding benchmark, impressive for its size at the time. Fair caveat: it was a narrow Python model, not a general assistant, and benchmark scores have their own problems (see Part 10).

The lesson: the synthetic data was designed for diversity and quality, then mixed with filtered real code. Nobody was looping a model on its own output and hoping.

3. Filtered self-generated data: keep only what passes a check

STaR (“Self-Taught Reasoner”, Zelikman and colleagues, 2022) is a beautiful small idea. The model writes step-by-step reasoning for many questions. You keep only the reasoning that led to the correct answer. You fine-tune on those. Repeat. For questions it got wrong, it’s shown the right answer and asked to explain its way there.

The model is learning from its own writing, but a filter outside the model (the known correct answer) decides what survives. That filter is what injects new information. Many modern reasoning-model training recipes rely on a similar idea: generate lots of attempts, check them against something objective like a unit test or a math answer, and learn from what passes.

4. Synthetic data for alignment, at scale

When Nvidia released Nemotron-4 340B in June 2024, its technical report said over 98% of the data used to align the model (the fine-tuning that turns a base model into a helpful assistant) was synthetically generated. Nvidia also published the pipeline. That pipeline leaned on a reward model to score and filter outputs.

The pattern

Put those four side by side and the rule is obvious:

Synthetic data works when something outside the loop adds information. A stronger teacher. A correct answer. A unit test. A reward model. A human-designed curriculum for diversity. Plus real data kept in the mix.

Synthetic data fails when the loop is closed. Same model, its own unfiltered output, original data thrown away.

The jackrabbits came from the second setup. The successful models came from the first.

“Are we running out of human text?”

This is the other half of why synthetic data matters so much.

In June 2024, the research group Epoch AI published an estimate of how much public, human-written text exists that’s good enough to train on. Their central figure: about 300 trillion tokens (with a wide 90% range of 100 to 1,000 trillion), after filtering for quality and allowing for training on the same data more than once.

At the rate training datasets were growing, they projected that frontier models would fully use that stock sometime between 2026 and 2032. If labs “overtrain” (train smaller models on far more data than usual, which is common because smaller models are cheaper to run), it could come sooner.

A few honest caveats, which Epoch itself raised:

The estimate covers public text only. It leaves out images, video, audio and private data like messages and documents.
Their 2022 estimate had predicted exhaustion around 2024. They pushed it later after learning that filtered web data was better than expected and that repeating data costs less than feared. Forecasts like this move.
The authors said themselves that keeping progress going beyond about 2030 would likely need new approaches: synthetic data, learning from other kinds of data, and better data efficiency.

At the time of writing (October 2026) we’re inside that projected window. I haven’t seen an updated public estimate I can verify, so treat the 2024 number as a dated estimate, not a countdown clock.

What the labs did in response is visible in public: more synthetic data with filters (above), more reinforcement learning on tasks with checkable answers, more video and other modalities, and paying for human data. In February 2024, Reuters reported that Google had agreed to license Reddit’s content for AI training in a deal worth about $60 million a year. Neither company disclosed the terms, so that figure is from press reporting. But the direction is clear: fresh human text has a price now.

What it means for the open web

Here’s where it gets personal for anyone who writes online.

In October 2025, Axios reported on a study by the SEO company Graphite. It sampled 65,000 English-language article URLs from Common Crawl (a huge public web archive that’s a common source of AI training data) and ran them through an AI-text detector. The finding: AI-generated articles roughly caught up with human-written ones around late 2024, then the share levelled off at about half. The same study found that human-written pages still dominated Google’s top results.

Treat those numbers carefully. It’s a vendor study, not peer-reviewed, and AI detectors make mistakes. But even a rough “a big share of new articles is machine-written” is enough to see the problem.

Three things follow:

Older human data gets more valuable. Text written before the AI boom is now a known, fairly clean resource. The Shumailov paper made this point directly: data about genuine human interactions will become increasingly valuable. Expect more licensing deals, more locked-down sites, and more fights about scraping.
Provenance becomes a real engineering problem. If you’re building a training set from the web today, you need to know (or estimate) what was written by people, by models, or by both. Labs filter heavily for quality. Low-effort AI spam tends to be low quality, so a good quality filter catches some of it anyway. Fluent, decent AI text is harder to separate.
Real human writing is a public good. Honest forum answers, personal experience, niche expertise, local knowledge, odd hobbies. Those are exactly the “tails” that collapse eats first. If the web fills up with average text, the rare, specific, human stuff is what future models will be missing.

That last point is why I still think writing about your own real experience, in your own words, is worth doing. Not because it’s good for SEO. Because it’s the part of the distribution machines can’t make up.

Limits and open questions

To be fair to both sides:

The scary experiments are deliberately extreme. Full replacement over many generations is a stress test. Real pipelines accumulate, filter and mix. The headline “AI is poisoning itself” overstates the risk for well-run labs.
But “accumulate” isn’t a free pass. It shows error stays bounded, not that rare knowledge is perfectly kept. If 90% of future data is synthetic, the tails can still get thin.
Detection is hard. Nobody can reliably label all AI-written text on the web, so “keep the real data” is easier to say than to do for new data.
Quality filters cut both ways. Filtering for “good” text can itself trim the tails, because unusual writing can look like low quality.
The big labs don’t publish full data recipes. We know synthetic data is widely used. We usually don’t know exact ratios.

What it means in practice

If you build with AI:

Never throw away your real data. If you fine-tune a model on its own outputs, or on outputs from another model, keep a solid share of real, human examples in every round.
Put a check in the loop. Tests that pass, answers that match, human review, a separate scoring model. Unfiltered self-generated data is how you get jackrabbits.
Measure diversity, not just accuracy. Collapse shows up first as sameness. Track whether outputs are getting more repetitive, and test on rare cases specifically, not just the average one.
Label where your data came from. Keep a field for “human”, “model-generated” and “model-generated, human-reviewed”. Future you will be glad.

If you buy AI:

Ask how a vendor’s custom model was trained, and whether synthetic data was used. “Yes, with filtering and real data mixed in” is a fine answer. “We generated it all with the model itself” deserves follow-up questions.
Test it on your unusual cases, not just your common ones. That’s where any narrowing shows up first.

Key ideas

Model collapse is what happens when models train repeatedly on their own outputs: rare cases (the tails) vanish first, then variety shrinks toward one bland answer.
It comes from small errors (sampling, capacity, training) that compound because each generation treats the last one’s mistakes as truth.
Keeping real data in the mix largely prevents it. Accumulating synthetic data alongside real data kept error bounded in Gerstgrasser et al. 2024; keeping 10% real data cut the damage in Shumailov et al.
Labs use synthetic data successfully when something outside the loop adds information: a stronger teacher (distillation), a correctness check (STaR-style filtering), a reward model, or a designed curriculum.
Epoch AI’s 2024 estimate put public human text at about 300 trillion usable tokens, likely fully used between 2026 and 2032. That’s a dated forecast, not a cliff.
Genuine human writing, especially the rare and specific kind, is getting more valuable, both to AI labs and to the web.

Sources: Shumailov et al., “AI models collapse when trained on recursively generated data”, Nature, 24 July 2024 (full text via PMC); Shumailov et al., “The Curse of Recursion”, arXiv, May 2023; Gerstgrasser et al., “Is Model Collapse Inevitable?”, arXiv, April 2024; Alemohammad et al., “Self-Consuming Generative Models Go MAD”, arXiv, July 2023; DeepSeek-AI, “DeepSeek-R1”, arXiv, January 2025; Gunasekar et al., “Textbooks Are All You Need”, arXiv, June 2023; Zelikman et al., “STaR: Bootstrapping Reasoning With Reasoning”, arXiv, 2022; Nvidia, “Nemotron-4 340B Technical Report”, June 2024; Epoch AI, “Will we run out of data?”, June 2024; Reuters via Yahoo, Reddit and Google licensing deal, February 2024; Axios on the Graphite study, October 2025.