Bet #4: Just Make It Bigger (Scale)

On March 13, 2019, Rich Sutton posted a short essay on his personal website. It has no figures and no equations. It fits on about two printed pages. It’s called “The Bitter Lesson”.

Sutton is one of the founders of modern reinforcement learning, the subject of the previous part. His essay opens with a claim about the whole history of the field: over 70 years, “general methods that leverage computation are ultimately the most effective, and by a large margin.”

His examples were chess, Go, speech recognition and computer vision. Each time, researchers spent years building human knowledge into their systems: how grandmasters think, how sounds form words, what edges and shapes look like. Each time, a simpler method that just used more computation, through search or learning, eventually beat them. The lesson was “bitter” because it meant a lot of clever human work had been wasted.

Sutton wasn’t writing about chatbots. But his essay became the unofficial manifesto for the biggest bet in AI history: take a fairly simple learning method, feed it more data and more computing power, and keep going. No new theory of intelligence. Just bigger.

Then, in September 2025, Sutton went on a podcast and said, in effect, that large language models were not the bitter lesson he meant. We’ll get to that. First, the bet.

The idea: intelligence as a resource problem

Earlier bets in this series each had a theory of what intelligence is: logic and rules, or maximising reward. The scaling bet mostly doesn’t care. It says: we already have a learning method that works. The bottleneck is resources.

The method is the neural network from Part 3, specifically the transformer (the architecture behind ChatGPT, from a 2017 Google paper), trained to predict the next word in a huge pile of text. The resources are three things:

• Parameters: the adjustable numbers inside the network. More parameters means a bigger model that can store more patterns.
• Data: how much text the model reads during training, counted in tokens (word pieces).
• Compute: the total number of calculations used to train it, measured in FLOP (floating-point operations, basically individual multiplications and additions).

The scaling hypothesis says that if you grow all three together, the model gets steadily better, and keeps getting better in ways that look more and more like general intelligence.

The clearest statement of the strong version came from the writer Gwern Branwen, in an essay called “The Scaling Hypothesis” published in May 2020, right after GPT-3 appeared. His argument: neural networks absorb data and compute, generalise better as problems get harder, and show new abilities as they grow. If that’s true, general intelligence might be simple learning applied at a scale we just hadn’t reached yet. Inside the labs, the people who acted on this belief most aggressively were at OpenAI, including its then chief scientist Ilya Sutskever, who had co-authored the 2012 AlexNet paper that kicked off the deep learning era.

What was built: GPT-1 to GPT-4

OpenAI’s GPT series is the cleanest record of the bet, at least until the numbers stopped being published.

• GPT-1 (June 2018): about 117 million parameters. A proof of concept that pre-training on lots of unlabelled text, then fine-tuning, worked well.
• GPT-2 (February 2019): 1.5 billion parameters in its largest form. Around 13 times bigger. It wrote surprisingly fluent paragraphs, and OpenAI initially held back the full model over misuse concerns, releasing it in stages through November 2019.
• GPT-3 (May 2020): 175 billion parameters. Over 100 times bigger than GPT-2. The paper’s title was “Language Models are Few-Shot Learners”: you could show GPT-3 a few examples of a task inside the prompt and it would often do the task, with no retraining at all.
• GPT-4 (March 2023): OpenAI didn’t disclose its size or training compute. Its technical report says only that it’s a transformer pre-trained to predict the next token. Any parameter count you’ve seen for GPT-4 is an outside estimate or a leak, not a published number.

Each jump was mostly the same recipe, scaled up. The surprise of 2018 to 2023 was how far one recipe went.

At an event at MIT in April 2023, Sam Altman was asked whether training GPT-4 cost $100 million. He replied: “It’s more than that.” In the same appearance, Wired reported him saying we were at the end of the era of “giant, giant models”, and that OpenAI would make models better in other ways. That remark turned out to be early, not wrong. Hold on to it.

Scaling laws: the bet gets a formula

A bet becomes an industry when you can predict the payoff. That happened in January 2020.

Jared Kaplan, Sam McCandlish and colleagues at OpenAI published “Scaling Laws for Neural Language Models”. They trained many language models of different sizes and measured their loss (how wrong the model’s next-word predictions are, on average). They found that loss falls along a smooth power law as you increase parameters, data or compute. Some of these trends held across more than seven orders of magnitude, meaning a range of over ten million times.

A power law is a straight line on a log-log chart. In plain terms: every time you multiply compute by ten, the error drops by a predictable fraction. A steady, boring, bankable improvement.

Kaplan’s paper also found that architecture details like the network’s width or depth mattered much less than raw scale, and that bigger models were more sample-efficient: they learned more from each token. So its advice was to spend most of a budget on a very large model and not train it for very long.

That advice was corrected two years later.

Chinchilla: you’re doing it wrong

In March 2022, Jordan Hoffmann and colleagues at DeepMind published “Training Compute-Optimal Large Language Models”. They trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. Their conclusion: for a fixed compute budget, model size and training data should grow in equal proportion. Double the parameters, double the tokens.

By that standard, the giant models of the time were undertrained. To prove it, they built Chinchilla: 70 billion parameters, trained on about four times more data than DeepMind’s own 280-billion-parameter Gopher, using the same compute. Chinchilla beat Gopher, GPT-3 (175B) and Megatron-Turing NLG (530B) on a wide range of tasks. On MMLU, a broad knowledge quiz, it scored 67.5%, more than 7 points above Gopher.

This mattered for two reasons. First, it made the scaling bet more efficient: same money, better model. Second, and more ominously, it made data a first-class bottleneck. If every doubling of model size needs a doubling of text, sooner or later you run out of text.

The compute curve

The labs took the scaling laws seriously. Here’s what happened to training compute.

Epoch AI, a research group that tracks these numbers, estimates that training compute for frontier language models has grown about 5 times per year since 2020, according to its trends page updated in February 2026. Some rough points from its database:

• AlexNet (2012): about 5 × 10¹⁷ FLOP.
• GPT-3 (2020): about 3 × 10²³ FLOP. Nearly a million times AlexNet.
• GPT-4 (2023): Epoch estimates about 2 × 10²⁵ FLOP, with a wide uncertainty range, since OpenAI didn’t publish it.
• GPT-4.5 (February 2025): Epoch estimates about 4 × 10²⁶ FLOP. OpenAI called it its largest model at the time.

Every gridline on that chart is ten times more computation. Going from AlexNet to the frontier in 2026 is a little over nine gridlines: billions of times more compute in 14 years.

The open circle is worth noticing. DeepSeek-V3, released in December 2024 by a Chinese lab, used an estimated 3 × 10²⁴ FLOP, about six times less than Epoch’s GPT-4 estimate, while DeepSeek reported benchmark results competitive with the leading models of the time. Efficiency gains are part of the scaling story too. The curve isn’t only about spending more. It’s also about getting more from each FLOP.

Emergent abilities, and the mirage

The most exciting claim of the scaling camp wasn’t the smooth loss curve. It was that new abilities appear suddenly as models grow.

In June 2022, Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel and colleagues published “Emergent Abilities of Large Language Models”. Their definition: an ability is emergent if it’s “not present in smaller models but is present in larger models.” They collected examples where performance on a task sat near random chance across many model sizes, then shot up past some threshold. Multi-step arithmetic. Certain word puzzles. Following instructions.

If abilities appear unpredictably with scale, a bigger model might suddenly do things nobody planned for. Exciting if you’re betting on scale, worrying if you work on safety.

In April 2023, Rylan Schaeffer, Brando Miranda and Sanmi Koyejo at Stanford published a reply with a pointed title: “Are Emergent Abilities of Large Language Models a Mirage?” Their argument is simple and clever. Many “emergent” results used all-or-nothing metrics, like exact match: an answer counts only if every digit is right. If a model gets steadily better at each digit, its exact-match score can still look flat for a long time and then jump. Measure the same models with a continuous metric, like how many digits are right, and the sudden jump often turns into a smooth slope.

So the “emergence” was partly in the ruler, not the model. The paper won an Outstanding Paper award at NeurIPS 2023, one of the field’s top conferences.

What’s fair to both sides: Schaeffer and colleagues showed many emergence claims depend on the metric. They didn’t show that bigger models aren’t better. And for users, all-or-nothing is often what matters: a calculation that’s 90% right is still wrong. The practical lesson is that capabilities do grow with scale, but you can’t reliably predict when a specific skill will become useful from the smooth curve alone.

The wall: late 2024

For four years, the bet paid off on schedule. Then, in late 2024, the mood changed.

In November 2024, The Information reported that OpenAI’s next flagship model, codenamed Orion, showed a smaller improvement over GPT-4 than GPT-4 had shown over GPT-3, according to some employees who had tested it. The report pointed to a shortage of high-quality training data as one reason. OpenAI pushed back publicly, and the report was based on anonymous sources, so treat it as a report, not a fact. But it captured a worry many people in the field shared.

Then came the line everyone quoted. On December 13, 2024, Ilya Sutskever, by then running his own company, Safe Superintelligence, gave a talk at NeurIPS in Vancouver. He said “pre-training as we know it will unquestionably end.”

His reasoning, as reported by Reuters and others: compute keeps growing, through better chips, better algorithms and bigger clusters. But data doesn’t, because “we have but one internet”. He compared training data to fossil fuel: a resource built up once and now being used up. Coverage summarised it as “peak data”. He also predicted what would come next: agents, synthetic data and more computation at inference time, and he warned that systems that reason more become less predictable.

This was striking because Sutskever had been one of the strongest believers in scale. He wasn’t saying scale was over. He was saying one specific kind of scale, pre-training on human-written text, was hitting its limit.

The data numbers support the worry. A 2024 Epoch AI paper led by Pablo Villalobos estimated the stock of usable public human text at roughly 300 trillion tokens, and projected that, at the trends of the time, models would use it all up somewhere between 2026 and 2032. That’s a forecast with wide error bars, not a deadline. Labs have responses (synthetic data, other modalities like video, better data efficiency), and I covered the risks of training on AI-made data in Synthetic Data and Model Collapse.

GPT-4.5, released on February 27, 2025, was in a sense the test case. It was OpenAI’s largest model, and by Epoch’s estimate used around 20 times GPT-4’s training compute. It was better, especially on knowledge and writing. But the original system card reportedly described it as not a frontier model, and on hard maths and coding it trailed OpenAI’s smaller “reasoning” models. The biggest pre-trained model was no longer obviously the smartest one.

The shift: scaling somewhere else

Here’s the part that’s easy to miss in the “scaling is dead” headlines. The bet didn’t die. It moved.

On September 12, 2024, OpenAI released o1, its first “reasoning model”. Its announcement reported that performance improved consistently with more reinforcement learning (train-time compute) and with more time spent thinking before answering (test-time compute, also called inference-time compute). In other words, o1 came with new scaling curves, on new axes.

That’s the subject of the next part. The three knobs:

1. Pre-training: the classic scaling laws. Still valuable, but increasingly limited by data.
2. Post-training: everything after pre-training, especially reinforcement learning on tasks where answers can be checked, like maths and code. Labs started spending far more compute here. When xAI launched Grok 4 in July 2025, Epoch’s reading of a launch slide was that its reinforcement learning compute was comparable to its pre-training compute, though that’s an outside interpretation of a vague claim.
3. Inference time: letting the model generate long chains of reasoning before answering. This compute is spent on every question, which changes the economics completely.

The third knob is the big change for anyone who builds products. In the old world, a model’s cost was mostly paid once, in training. In the new one, a hard question can cost far more than an easy one, because the model “thinks” longer. Altman’s 2023 line about making models better “in other ways” turned out to describe the next two years quite well.

Cost and energy

Scaling isn’t free, and the bill has grown as fast as the compute.

• Training cost. A 2024 Epoch AI paper by Ben Cottier and colleagues estimated that the amortised cost of training the most compute-intensive models grew about 2.4 times per year since 2016, and projected that the largest training runs would exceed $1 billion by 2027 if the trend held. Epoch’s trends page puts frontier language model training cost growth at about 3.5 times per year since 2020.
• Data centres. Epoch noted in early 2026 that the largest AI data centres were approaching a gigawatt of power capacity, roughly the output of a large power plant.
• Electricity. The International Energy Agency’s “Energy and AI” report (April 2025) estimated that data centres used about 415 terawatt-hours in 2024, around 1.5% of the world’s electricity, and projected about 945 TWh by 2030 in its base case, slightly more than Japan uses today. That’s all data centres, not just AI, but the IEA named AI as the main driver of growth.

Each 10× in compute needs more chips, buildings, power lines and money. The question is no longer just “does scale work?” but “who can afford the next 10×, and will the returns justify it?” I wrote about why all of this runs on specialised chips in Why AI Runs on GPUs.

Two founders of the bet, second thoughts

In 2025, two people closely linked to the scaling bet publicly questioned how far it goes.

Rich Sutton, on Dwarkesh Patel’s podcast in September 2025, said it was “an interesting question” whether large language models are a case of the bitter lesson at all. His point: LLMs learn by imitating human text, which is itself a huge injection of human knowledge, exactly what his essay warned against. In his words, LLMs can “predict what a person would say,” but not what will happen in the world, and they don’t learn continually from their own experience. He expects systems that learn from experience to be the real next instance of the bitter lesson. Many listeners pushed back, arguing that LLMs can be the foundation that experiential learning builds on. That debate is open.

Ilya Sutskever, on the same podcast in November 2025, divided recent history into eras. From 2012 to 2020 was “the age of research”. From 2020 to 2025 was “the age of scaling”, with error bars on the years. Now, he argued, it’s back to research, “just with big computers”. Asked whether 100 times more scale would transform everything, he said he didn’t think so, though results would be different. His main concern was generalisation: models ace benchmarks yet still fail in ways a person wouldn’t.

Neither of them said scale is useless. Both said scale alone isn’t the whole answer. Coming from these two, that’s a meaningful shift.

The other side deserves a fair hearing too. Every previous “scaling is hitting a wall” moment, before 2024, was followed by models that got better. Labs kept building bigger clusters. And the move to reasoning models produced some of the fastest benchmark gains the field had ever seen. A scaling optimist in 2026 would say the bet simply found new places to put the compute.

October 2026: where scale stands

Here’s what can be said with reasonable confidence, at the time of writing.

• Compute is still growing. Epoch AI’s database lists OpenAI’s GPT-6 Astra, released in September 2026, as the largest training run it has estimated, at roughly 9 × 10²⁶ FLOP, with a range of about 5 × 10²⁶ to 2 × 10²⁷. That estimate is based on reports that it was trained on over 100,000 GPUs. OpenAI hasn’t published the figure.
• Parameter counts are mostly secret. None of the leading closed labs publish the size of their frontier models anymore. The public story of “bigger and bigger numbers” ended with GPT-3.
• Pre-training alone is no longer the main lever. The leading labs now scale post-training and inference-time compute alongside pre-training. Which of the three contributes most to each new model isn’t publicly known.
• The physical limits are now power and money, not just algorithms. The bets being placed in 2026 are measured in gigawatts.

Did “just make it bigger” get us to AGI? Not by itself, by the accounts of two of its own founders. Did it fail? Also no. It produced the most capable AI systems ever built, and it’s still the backbone of every frontier model. The pure bet hit a wall. The broader bet, that computation beats cleverness in the long run, is still very much alive. It just learned that what you spend the computation on matters.

Scoreboard

Key ideas

• The Bitter Lesson (Sutton, 2019): in the long run, general methods that use more computation beat methods built on human knowledge. It became the scaling camp’s manifesto.
• Scaling laws (Kaplan, 2020) showed model error falls predictably as parameters, data and compute grow. Chinchilla (2022) showed parameters and data should grow together, which made data a bottleneck.
• Emergent abilities looked like sudden jumps with scale. The “mirage” critique showed many jumps came from all-or-nothing metrics, though bigger models really were better.
• Late 2024 brought reports of diminishing returns from pre-training and Sutskever’s “peak data” warning.
• The bet moved, not died: since o1 (2024), labs scale reinforcement learning and inference-time thinking, not just pre-training.
• Cost and energy are now the real limits: training costs growing 2.4× to 3.5× per year by Epoch’s estimates, and data centre electricity projected to roughly double by 2030 (IEA).

Sources: Sutton, “The Bitter Lesson” (March 13, 2019); Gwern Branwen, “The Scaling Hypothesis” (2020); Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2, 2019); Brown et al., “Language Models are Few-Shot Learners” (GPT-3, 2020); OpenAI, “GPT-2: 1.5B release” (November 2019); OpenAI, “GPT-4 Technical Report” (2023); Wired, “OpenAI’s CEO Says the Age of Giant AI Models Is Already Over” (April 2023); Kaplan et al., “Scaling Laws for Neural Language Models” (2020); Hoffmann et al., “Training Compute-Optimal Large Language Models” (2022); Wei et al., “Emergent Abilities of Large Language Models” (2022); Schaeffer, Miranda and Koyejo, “Are Emergent Abilities of Large Language Models a Mirage?” (2023) and NeurIPS 2023 awards; Epoch AI, Trends (updated February 2026) and Notable AI Models database (accessed October 2026); Villalobos et al., “Will we run out of data?” (2024); Cottier et al., “The rising costs of training frontier AI models” (2024); The Decoder on The Information’s report, “OpenAI’s new Orion model reportedly shows small gains over GPT-4” (November 2024); Computing, report on Sutskever’s NeurIPS 2024 talk (December 2024) and Reuters via The Standard; OpenAI, “Learning to Reason with LLMs” (September 2024) and “Introducing GPT-4.5” (February 2025); Interconnects, “GPT-4.5: Not a frontier model?” (2025); IEA, “Energy and AI” (April 2025); Dwarkesh Patel, interviews with Richard Sutton (September 2025) and Ilya Sutskever (November 2025); Constellation Research, “OpenAI launches GPT-6 Astra” (September 2026).

Related: Bet #3: Reward Is Enough (Reinforcement Learning), Bet #5: Teach It to Think (Reasoning and Agents), and Why AI Runs on GPUs.