Bet #3: Reward Is Enough (Reinforcement Learning)

In March 2016, in a hotel in Seoul, a computer played a move that experts thought was a mistake.

It was the second game of a five-game Go match between Lee Sedol, one of the strongest players of his generation, and AlphaGo, a program built by DeepMind. On its 37th move, AlphaGo placed a stone on the fifth line from the edge of the board, in a spot that centuries of Go wisdom said was too loose and too early. Commentators were puzzled. Some assumed the machine had glitched.

It hadn’t. AlphaGo’s own estimate, DeepMind said later, was that a human player would choose that move about 1 in 10,000 times. It played it anyway, because its own experience said it was good. It went on to win the game. It won the match 4 to 1. DeepMind says over 200 million people watched.

Move 37 became a symbol of something new: a machine that didn’t just copy human expertise, but found ideas humans hadn’t. And the way it found them was the oldest trick in learning: try things, see what pays off, do more of that.

That’s the third bet. Part 3 was about how a machine learns: neural networks. This part is about what it learns from. The reinforcement learning bet says you don’t need to hand the machine rules, or even examples of correct answers. Give it a goal, a way to act and a score. Reward is enough.

The idea: learning from consequences

Reinforcement learning, or RL, is learning by trial and error.

There is an agent (the learner) and an environment (the world it acts in, which might be a game, a simulator or a real robot’s surroundings). At each step, the agent sees the current state, picks an action, and gets back a new state and a reward, a number that says how good that was. The agent’s job is to learn a policy, a way of choosing actions, that collects as much reward as possible over time.

Notice what’s missing. Nobody tells the agent the right move. It only learns, after the fact, how things turned out. In Go, the reward might arrive only at the end of the game, hundreds of moves later. Which of those moves deserved the credit? That’s the same credit assignment problem from Part 3, stretched out over time.

The field’s textbook, Richard Sutton and Andrew Barto’s Reinforcement Learning: An Introduction, first came out in 1998. In March 2025, the two received the 2024 Turing Award “for developing the conceptual and algorithmic foundations of reinforcement learning”. But the story starts long before them.

1959: Samuel’s checkers

In the 1950s, Arthur Samuel, an engineer at IBM, wrote a program to play checkers (draughts). It ran on IBM’s early computers, starting with the IBM 701.

His 1959 paper, “Some Studies in Machine Learning Using the Game of Checkers”, reported that a computer could learn to play better than the person who programmed it, given only the rules, a sense of direction and some loosely specified parameters, in 8 to 10 hours of playing time. Much of that learning came from the program playing against itself and adjusting how it scored board positions. IBM credits Samuel with popularising the phrase machine learning.

In 1962, the program beat Robert Nealey, a self-described checkers master, in a single game. It was a famous moment. It was also oversold. Nealey’s tournament record didn’t really support the “master” label, and in a rematch a year later he beat the program without losing a game.

The pattern, an honest real result followed by a bigger story, will repeat.

1992: TD-Gammon teaches backgammon players something new

The next landmark came from Gerald Tesauro at IBM, with backgammon.

Tesauro combined a neural network with a method called temporal difference (TD) learning, developed by Sutton in the 1980s. The idea of TD learning: you don’t wait until the end of the game to learn. After each move, you compare your prediction of how the game is going with your prediction one move later, and adjust towards the newer one. The network learns to predict, and the predictions get sharper as the game goes on.

TD-Gammon learned by playing against itself, starting from random play. Tesauro’s 1995 account in Communications of the ACM reports the results against people:

• Version 1.0, in late 1991, played 51 games against three strong players, including Bill Robertie, and lost by a net 13 points, about a quarter of a point per game.
• After further versions, Robertie judged version 2.1 to be at a strong master level, close to the world’s best.

The most striking result wasn’t the score. TD-Gammon changed how top humans played. For several common opening rolls, experts had long favoured one move. TD-Gammon’s analysis suggested a different one was better. Top players tried it, it worked, and Tesauro reported that the old move had virtually disappeared from tournament play.

A small move 37, 24 years early.

2013–2015: one network, 49 Atari games

The modern era of RL started with old video games.

In 2013, a small London company called DeepMind posted a paper called “Playing Atari with Deep Reinforcement Learning”. In February 2015, an expanded version appeared in Nature as “Human-level control through deep reinforcement learning”, by Volodymyr Mnih and colleagues.

The system, called DQN (Deep Q-Network), joined the two bets. It used a deep neural network from Part 3 to look at raw screen pixels, and reinforcement learning to decide which joystick action to take. It was given only the pixels and the score. It had no idea what a paddle, a ball or an alien was.

The same network design and settings were used on 49 different Atari 2600 games. The paper reports that DQN beat the best previous RL methods on 43 of them, and scored more than 75 percent of a professional human games tester’s score on 29.

Two details from the paper matter for later. First, each game was trained for 50 million frames, which the authors describe as about 38 days of game experience. Second, games that need long-term planning, like Montezuma’s Revenge, where you must do many things before scoring anything, “still constitute a major challenge”.

2016: AlphaGo

Go was the obvious next target. It has far more possible positions than chess, and before 2015 the best programs were nowhere near top professionals.

AlphaGo combined three things:

• A neural network first trained to predict moves from records of human games.
• Reinforcement learning by playing against itself, to improve beyond those human moves.
• A search method (Monte Carlo tree search) that looks ahead through possible continuations, guided by the networks.

In October 2015, it beat the European champion Fan Hui 5 to 0, the first time an AI system had beaten a Go professional. In March 2016 came Lee Sedol. After AlphaGo’s move 37 in game two, Lee answered in game four with his own move 78, which DeepMind later said AlphaGo had also rated around 1 in 10,000. It won Lee that game. It was the only game a human took from AlphaGo in that match.

AlphaGo Zero, AlphaZero, MuZero: taking the humans out

DeepMind then removed the human parts, one by one.

AlphaGo Zero (Nature, October 2017) started from random play with no human games at all. Just the rules and self-play. DeepMind reported that after three days of training it beat the version that had played Lee Sedol, 100 games to 0, and after about 40 days it surpassed every previous version. It ran on 4 of Google’s TPU chips, against 48 for the Lee Sedol version.

AlphaZero (Science, December 2018) used one algorithm for three games: Go, chess and shogi (Japanese chess). DeepMind reported that it outperformed Stockfish, the reigning computer chess champion, after just 4 hours of training. In a 1,000-game match it won 155 games and lost 6, with the rest drawn.

MuZero (preprint November 2019, Nature December 2020) wasn’t even given the rules. It learned its own internal model of how the game works, and planned with that. It matched AlphaZero at Go, chess and shogi, and set a new state of the art across 57 Atari games. (Learned models of the world are their own bet. See World Models.)

AlphaStar and OpenAI Five: messy, real-time games

Board games are tidy: you see the whole board and take turns. StarCraft II and Dota 2 have hidden information, many units and decisions every fraction of a second.

AlphaStar (DeepMind). In January 2019, DeepMind showed AlphaStar beating two professional StarCraft II players, Dario “TLO” Wünsch and “MaNa”, 5 to 0 each in recorded matches, then losing a live game to MaNa. Press reports said each agent had trained on the equivalent of more than 200 years of play. Critics pointed out that those early versions saw the map through a zoomed-out “raw” view rather than moving a camera like a human, and the version that lost the live game was the one that had to use a camera. By October 2019, a version with tighter limits on its camera view and action speed reached Grandmaster level for all three races, ranked above 99.8 percent of active players on the official Battle.net server (Nature, 2019). Note that AlphaStar started from agents trained to imitate human games, then improved with RL.

OpenAI Five (OpenAI). On 13 April 2019, a team of five AI agents beat OG, the reigning Dota 2 world champions, 2 games to 0. OpenAI said this version had accumulated about 45,000 years of self-play over roughly ten real months. The games used a restricted pool of 17 heroes out of more than 100. In a public online event that month, OpenAI reported the bots won 7,215 competitive games against human teams and lost 42.

By 2019, RL had beaten top humans at the hardest games it had seriously tried. What was left?

“Reward is enough” (2021)

In 2021, four of the field’s leaders, David Silver, Satinder Singh, Doina Precup and Richard Sutton, put the bet in writing. Their paper, “Reward is enough”, appeared in the journal Artificial Intelligence.

Their hypothesis: intelligence, and the abilities that go with it, can be understood as serving the maximisation of reward by an agent acting in its environment. Knowledge, learning, perception, social intelligence, language, generalisation, imitation: they argued each could emerge from an agent trying to get more reward in a rich enough world. They suggested that powerful RL agents could be a path to artificial general intelligence.

The most direct reply came from Peter Vamplew and eleven colleagues, “Scalar reward is not enough” (2022). Their point: real goals are rarely one number. A self-driving car wants to arrive quickly and safely and legally. Squashing those into a single score hides trade-offs, and they argued that building general intelligence around one scalar reward carries unacceptable safety risks.

A second line of criticism is practical. Even if reward is enough in principle, someone still has to write the reward down, and agents are very good at finding gaps between what you scored and what you meant. That problem has a name and a long list of real examples: reward hacking.

Gato (2022): one agent, many tasks

If reward is the road to general intelligence, you’d want one agent that does many things, not a different champion for every game.

In May 2022, DeepMind published “A Generalist Agent”, describing Gato. A single network of about 1.2 billion parameters was trained on 604 distinct tasks: Atari games, captioning images, chatting, stacking blocks with a real robot arm. The paper reports that Gato performed above 50 percent of the expert score on more than 450 of the 604 control tasks.

It was a real step towards one model doing many things. But Gato was trained by imitating data, much of it produced by specialist RL agents, rather than by RL itself, and its scores were measured as a share of those specialists’ scores, not as wins over them. The model that actually took general-purpose AI forward that year was a different bet entirely: large language models (see Bet #4).

Why games didn’t generalise

So why didn’t superhuman Go turn into superhuman everything? Four reasons come up again and again.

1. Sample efficiency. RL agents need enormous amounts of experience. DQN used about 38 days of play per Atari game. AlphaStar used centuries of StarCraft. OpenAI Five used 45,000 years of Dota. A human gets good at Breakout in an afternoon. In a game, you can buy experience with compute. In the real world, you can’t run a factory or a hospital for 45,000 years.

2. Simulators. All of those years were possible because games are perfect simulators. They run faster than real time, in parallel, and reset instantly when you fail. A real robot breaks, a real customer leaves, and a real day takes a day. Building a simulator good enough to learn from is often harder than the task itself.

3. Sparse rewards. Many real tasks give no feedback until the very end, or none at all. Montezuma’s Revenge was the classic case: random exploration almost never stumbles on a reward, so there is nothing to learn from. DeepMind’s Agent57 finally beat the human benchmark on all 57 Atari games in March 2020, including Montezuma’s Revenge, but needed special exploration machinery and was still trained game by game.

4. Narrow, brittle mastery. Superhuman in one environment did not mean robust. In 2023, researchers including Tony Wang, Adam Gleave and Kellin Pelrine trained an “adversarial” program that won more than 97 percent of games against KataGo, an open-source Go AI far stronger than any human, by luring it into a blunder it didn’t understand. Pelrine, a strong amateur, then learned the trick himself and beat a top KataGo-based bot on a public Go server in 14 of 15 games. Superhuman, yes. Understanding Go the way a human does, no.

And underneath all four: real life has no score. Games come with a clear reward built in. The world doesn’t.

The comeback: RL inside language models

For a few years after 2019, it looked like RL had been sidelined. Then it came back, in a place few expected: inside chatbots.

RLHF: reward from human preferences

If you can’t write a reward down, maybe you can learn it from people. In 2017, Paul Christiano and colleagues at OpenAI and DeepMind showed that an agent could learn tasks from humans simply comparing pairs of short clips and picking the better one.

In 2022, OpenAI’s InstructGPT paper applied this to language models. People ranked model answers. A reward model, a second network, learned to predict those rankings. Then RL tuned the language model to get high scores from the reward model. This is RLHF, reinforcement learning from human feedback. The striking result: people preferred answers from a 1.3 billion parameter InstructGPT model over the original 175 billion parameter GPT-3, despite it being over 100 times smaller.

RLHF also brought the old RL problems with it. A learned reward model is a proxy, and models learned to flatter and to pad their answers (see Reward Hacking).

RL for reasoning: rewards you can check

The bigger comeback came in 2024 and 2025. The insight: in maths and code, you often don’t need a human judge. You can check the answer. Did the final number match? Did the tests pass? That gives a clean reward, like a game score.

• In September 2024, OpenAI released o1, a model trained with large-scale RL to “think” before answering. OpenAI reported that its performance “consistently improves with more reinforcement learning”.
• In January 2025, DeepSeek published DeepSeek-R1. Its R1-Zero variant was trained with RL directly on a base model, without human-written reasoning examples. On the AIME 2024 maths competition, its score (answering each problem once) rose from 15.6 percent to 71.0 percent during RL training. The paper was later peer-reviewed and published in Nature in September 2025.
• In July 2025, Google DeepMind reported that an advanced Gemini Deep Think model scored 35 out of 42 at the International Mathematical Olympiad, a gold-medal score confirmed by the IMO. DeepMind said it had trained the model with “novel reinforcement learning techniques” on multi-step reasoning and theorem-proving data.

It’s TD-Gammon and AlphaGo again, except the agent is a language model and the moves are words. Bet #5 picks up this story.

Status in October 2026

Where does “reward is enough” stand?

RL won a place in every frontier recipe. At the time of writing, OpenAI, Google DeepMind and DeepSeek have all publicly described RL as a key stage in training their reasoning models. RL is no longer a separate camp. It’s a layer on top of the other bets.

But the pure version of the bet hasn’t been tested. Today’s systems don’t learn everything from reward. They learn most of what they know from huge amounts of human-written text (Bet #4), then use RL to sharpen it. The reward is usually a checker, a test suite or a reward model trained on human preferences.

The founders want more. In April 2025, David Silver and Richard Sutton published an essay, “Welcome to the Era of Experience”, arguing that learning from human data is reaching its limits, and that the next leap will come from agents that learn mainly from their own experience in the world. In a September 2025 interview with Dwarkesh Patel, Sutton went further, arguing that large language models are a dead end because they can’t learn continually from experience on the job. Patel and many in the LLM camp disagree: they see language models as the foundation that experiential learning can build on.

Both sides agree on one thing. The four walls from the games era, sample efficiency, simulators, sparse rewards and brittle narrowness, haven’t gone away. They’ve moved. Verifiable rewards work well in maths and code. For tasks with no clear score, like running a business or doing research, nobody has a reward signal that can’t be gamed.

Scoreboard

Key ideas

• Reinforcement learning is trial and error with a score. An agent acts, sees what happens and gets a reward, and learns a policy that collects more reward over time.
• Self-play was the engine. From Samuel’s checkers to AlphaGo Zero, playing against yourself turned compute into experience, and sometimes into ideas humans hadn’t found, like move 37.
• Games were the perfect environment, with clear rules, a built-in score and unlimited fast replays. That’s why RL conquered them, and why it didn’t easily leave them.
• The four walls: sample efficiency, the need for simulators, sparse rewards and narrow, brittle mastery.
• “Reward is enough” (2021) is a hypothesis, not a result. Critics argue real goals aren’t one number, and that any written reward can be gamed.
• RL came back inside language models, first as RLHF, then as RL on checkable answers for reasoning. In 2026 it’s a layer on top of large pre-trained models, not a replacement for them.

Sources: DeepMind, AlphaGo (move 37 and move 78 probabilities, viewership, Fan Hui match) and British Go Association match report; Sutton and Barto, Reinforcement Learning: An Introduction (1998) and ACM, “2024 ACM A.M. Turing Award” (5 March 2025); Samuel, “Some Studies in Machine Learning Using the Game of Checkers” (IBM Journal, 1959), IBM, “The games that helped AI evolve”, and Chinook project, “Arthur Samuel’s Legacy” on the Nealey games; Tesauro, “Temporal Difference Learning and TD-Gammon” (Communications of the ACM, March 1995); Mnih et al., “Playing Atari with Deep Reinforcement Learning” (arXiv, 2013) and “Human-level control through deep reinforcement learning” (Nature 518, 2015); Silver et al., “Mastering the game of Go with deep neural networks and tree search” (Nature, 2016); DeepMind, “AlphaGo Zero: Starting from scratch” (18 October 2017); DeepMind, “AlphaZero: Shedding new light on chess, shogi, and Go” (6 December 2018); Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model” (Nature 588, 2020); Engadget on AlphaStar vs TLO and MaNa (January 2019) and DeepMind, “AlphaStar: Grandmaster level in StarCraft II” (30 October 2019), plus AI Impacts, “The unexpected difficulty of comparing AlphaStar to humans”; OpenAI, “OpenAI Five”, “OpenAI Five defeats Dota 2 world champions” (April 2019), Berner et al., “Dota 2 with Large Scale Deep Reinforcement Learning” (2019), and Engadget on the Arena results; Silver, Singh, Precup and Sutton, “Reward is enough” (Artificial Intelligence 299, 2021); Vamplew et al., “Scalar reward is not enough” (Autonomous Agents and Multi-Agent Systems, 2022); Reed et al., “A Generalist Agent” (2022); DeepMind, “Agent57: Outperforming the human Atari benchmark” (31 March 2020); Wang et al., “Adversarial Policies Beat Superhuman Go AIs” (ICML 2023) and FAR AI human evaluation; Christiano et al., “Deep reinforcement learning from human preferences” (2017); Ouyang et al., “Training language models to follow instructions with human feedback” (NeurIPS 2022); OpenAI, “Learning to Reason with LLMs” (12 September 2024); DeepSeek-AI, “DeepSeek-R1” (January 2025; Nature, September 2025); 9to5Google on Gemini Deep Think at IMO 2025 (21 July 2025); Silver and Sutton, “Welcome to the Era of Experience” (April 2025), via TechTalks; Dwarkesh Podcast, “Richard Sutton – Father of RL thinks LLMs are a dead end” (26 September 2025).

Related: Bet #2: Let It Learn (Neural Networks), Bet #4: Just Make It Bigger (Scale), and Reward Hacking: When AI Wins the Wrong Game.