Bet #5: Teach It to Think (Reasoning and Agents)

On December 20, 2024, the last day of OpenAI’s “12 Days of OpenAI” livestreams, Sam Altman and OpenAI’s research chief Mark Chen were joined by the people who run the ARC Prize. They had a score to announce.

ARC-AGI is a test of small visual puzzles. Each one shows a few examples of coloured grids being transformed, and you have to work out the rule and apply it to a new grid. Most people find them easy. AI had found them very hard. According to ARC Prize, the benchmark took four years to go from 0% with GPT-3 in 2020 to 5% with GPT-4o in 2024.

OpenAI’s new model, o3, scored 75.7% within the competition’s public compute limit. A high-compute version, using about 172 times more computation, scored 87.5%.

François Chollet, the creator of ARC-AGI and a long-time sceptic of claims that language models can reason, called it “a surprising and important step-function increase in AI capabilities.” In the same post, he added: “I don’t think o3 is AGI yet.”

Both sentences turned out to be right. This part is about the bet behind that moment: that the missing piece wasn’t a bigger model, but teaching the model to think before it answers.

The idea: thinking takes time

The previous part ended with the scaling bet hitting a wall in late 2024. Bigger pre-trained models were giving smaller gains, and good training text was running out.

The reasoning bet offers a different route. Humans don’t answer every question instantly. For a hard maths problem, you write things down, try an approach, notice a mistake and back up. Psychologists sometimes call these “fast” and “slow” thinking. A standard language model only has the fast kind: it produces each word in one pass through the network, with a fixed amount of computation per word, no matter how hard the question is.

The bet: give the model room to think in words, train it to use that room well, and let it spend more computation on harder problems. If that works, intelligence becomes partly a matter of how long you let the model think. And once a model can reason through steps, you can let it take actions too, which turns a chatbot into an agent: a system that plans, uses tools and works towards a goal over many steps. I explained what that word really means in What an AI Agent Actually Is.

Step 1: chain of thought (2022)

The first evidence came from prompting, not training.

In January 2022, Jason Wei and colleagues at Google published “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. The trick was simple. Instead of showing the model example questions with just answers, show it examples where the answer is worked out step by step. The model copies the pattern and writes out its own steps before answering.

With eight such worked examples, Google’s 540-billion-parameter PaLM model reached state-of-the-art accuracy on GSM8K, a benchmark of grade-school maths word problems, beating a GPT-3 model that had been specially fine-tuned for the task and paired with a checker.

Four months later, in May 2022, Takeshi Kojima and colleagues at the University of Tokyo and Google showed you didn’t even need examples. Adding one sentence to the prompt, “Let’s think step by step”, was enough. On OpenAI’s text-davinci-002 model, accuracy on MultiArith (a set of arithmetic word problems) rose from 17.7% to 78.7%. On GSM8K, it went from 10.4% to 40.7%.

One sentence, quadrupled accuracy. That told researchers something important: the ability was already in the model. It just needed space to work things out. The model’s own written steps act like a scratchpad, letting it break a hard problem into easy pieces, each solvable in one fast pass.

Step 2: train it to think (o1, 2024)

Prompting a model to think step by step is a nudge. The next question was whether you could train a model to reason well.

On September 12, 2024, OpenAI released o1, which it described as a model trained with reinforcement learning to think before answering. Before responding, o1 generates a long, mostly hidden chain of thought: it tries approaches, checks its work and corrects itself.

OpenAI’s reported results were a big jump. On AIME 2024, a hard US high-school maths competition, GPT-4o solved about 12% of problems on average. o1 solved 74% with one attempt, and 83% when taking the most common answer across 64 attempts. OpenAI also reported o1 placing in the 89th percentile on Codeforces competitive programming problems, and beating PhD-level experts’ accuracy on GPQA, a set of hard science questions. These are the lab’s own numbers, measured on its maximum thinking setting.

The most important line in the announcement wasn’t a score. OpenAI said o1’s performance improved consistently with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). That’s a new scaling law on a new axis. If pre-training was running short of data, here was a different place to put the compute.

Step 3: reinforcement learning, in the open (DeepSeek-R1, 2025)

OpenAI didn’t explain how o1 was trained. In January 2025, a Chinese lab did.

On January 20, 2025, DeepSeek released DeepSeek-R1, with a paper and openly downloadable model weights. The paper’s most striking result was a model called R1-Zero. DeepSeek took its base model and trained it with reinforcement learning on problems where the answer could be automatically checked, like maths with known answers and code with tests. No human-written examples of reasoning at all. Just a reward for getting the right answer, and a simple format rule.

The model taught itself to reason at length. On AIME 2024, R1-Zero’s single-attempt score rose during training from 15.6% to 71.0%, and to 86.7% with majority voting across attempts. The paper describes an “aha moment” at an intermediate checkpoint, where the model learned to stop, reconsider its first approach and spend more time on the problem, written in a strikingly human tone.

R1-Zero had problems: its reasoning was hard to read and mixed languages. So DeepSeek added a small amount of curated data and several training stages to make R1, which it reported performed comparably to OpenAI’s o1 on reasoning tasks.

This is the reinforcement learning bet coming back, on top of the scaling bet. Pre-training gives the model knowledge and language. RL on checkable problems teaches it to use them.

The release also shook the markets. On January 27, 2025, Nvidia’s share price fell 17% and it lost about $589 billion in market value, which Bloomberg reported as the largest one-day loss in US stock market history, as investors worried that strong AI might need fewer expensive chips. In September 2025, a version of the R1 paper was published in Nature after peer review, reporting that R1’s reasoning training cost about $294,000, on top of roughly $6 million for the base model. That’s a company-reported figure for compute, not the full cost of the research.

By mid-2025, every major lab had a reasoning model. The bet had become the standard.

The ARC-AGI scoreboard

ARC-AGI is the best place to watch this bet play out, because its designers keep making it harder on purpose.

ARC-AGI-1 and o3 (December 2024). The 75.7% run cost about $26 per task by ARC Prize’s current estimate. The 87.5% run cost about $4,560 per task, so roughly $456,000 for the 100 test tasks. ARC Prize has revised these figures as OpenAI’s pricing changed, but the order of magnitude is the point. Two other caveats: OpenAI said it had trained this o3 on 75% of ARC’s public training set, and the o3 OpenAI later released was a different version from the one tested.

ARC-AGI-2 (March 2025). Designed to be harder for reasoning systems while staying easy for people. At launch, ARC Prize reported that pure language models scored 0% and reasoning systems only single digits, with o3-preview at about 4%. Every task had been solved by at least two people in two attempts or fewer, and the average human tester got about 60%.

By the end of ARC Prize 2025, announced on December 5, 2025:

• The best Kaggle competition entry, NVARC, scored 24.03% on the private set at about $0.20 per task, under strict compute limits.
• The best verified commercial model, Claude Opus 4.5 with extended thinking, scored 37.6% at $2.20 per task.
• A “refinement” system by Poetiq, which wraps Gemini 3 Pro in a loop of generating, testing and improving answers, reached 54% at about $30 per task.

Then 2026 happened. On ARC Prize’s leaderboard data as of October 19, 2026, GPT-6 Astra scores 95.0% on ARC-AGI-2 at its maximum setting, at about $1.12 per task. Claude Opus 5.5 scores 93.3% at about $0.41 per task. The best open-weight model, Moonshot AI’s Kimi K3, scored 60.4% when ARC Prize verified it in July 2026. A test designed in early 2025 to stump reasoning models was close to solved within about 18 months, and for about a dollar a task.

ARC-AGI-3 (March 2026). So ARC Prize changed the format. ARC-AGI-3, launched on March 25, 2026, is a set of hundreds of small turn-based games with no instructions, no stated rules and no stated goal. An agent has to explore, work out what the game is, and then use that knowledge on harder levels. Humans scored 100% under ARC’s scoring. Frontier AI scored 0.51% at launch.

On September 3, 2026, ARC Prize reported that OpenAI’s GPT-6 Astra scored 62.7% with ARC’s standard setup, from its best observed run, at a cost of about $26,000 for the evaluation. With a different, OpenAI-specific harness (the software around the model) that lets it keep its hidden reasoning between turns, the same model scored 99.9% for about $19,000. ARC Prize lists these separately and says results from different harnesses aren’t directly comparable. It called Astra meaningful progress towards generalisation, and added: “we are not claiming that it is AGI.” As of its leaderboard data on October 19, 2026, 62.7% was still the top standard-setup score. How much of a score comes from the model and how much from the scaffolding around it is now a live question. I wrote about that wider problem in AI Benchmarks and Goodhart’s Law.

Humanity’s Last Exam

ARC tests puzzles that are easy for people. Humanity’s Last Exam tests the opposite: questions that are hard even for experts.

Released in January 2025 by the Center for AI Safety and Scale AI, with Dan Hendrycks and Alexandr Wang among the authors, it contains 2,500 questions written by subject experts across dozens of fields, from maths and physics to the humanities. Questions were kept only if top models of the time got them wrong. At launch, frontier models scored under 10%, and they were badly calibrated: they gave wrong answers with high confidence.

On Scale AI’s leaderboard, checked in October 2026, the top score is GPT-6 Astra at 54.8%, with a calibration error of 39%. That’s a huge jump in under two years, and still far from the ceiling. It also says something about the gap between knowing and reasoning: a model can be very strong on ARC-style puzzles and still miss nearly half of a hard expert exam, while often being confident about the wrong answers.

How long can an agent work on its own?

Benchmarks score single questions. Agents do jobs. In March 2025, METR, a non-profit that evaluates AI systems, proposed a way to measure that.

They gave AI agents a large set of software and research tasks, and also had skilled humans do them to measure how long each task takes a person. Then for each model they asked: what length of task can it complete about 50% of the time? They called this the “50% time horizon”.

The original paper, published on March 19, 2025, found the horizon had been doubling about every seven months for six years. Claude 3.7 Sonnet, from February 2025, could handle tasks of about an hour. METR noted that if the trend continued to the end of the decade, AI could carry out month-long projects on its own. They were careful to say that’s an extrapolation, not a prediction.

The trend has continued, and sped up. Using METR’s updated method (Time Horizon 1.1, from January 2026), the doubling time since 2023 is about 4 months. GPT-5 (August 2025) came in at about 3.4 hours. Claude Opus 4.6 (February 2026) at about 12 hours. An early version of Claude Mythos Preview, measured in April 2026, at about 17 hours.

The caveats matter as much as the line:

• The error bars are huge. METR’s 95% range for Claude Opus 4.6 runs from about 5 hours to about 60 hours.
• The test is running out. METR says measurements above 16 hours are unreliable with its current task suite, and it noted in March 2026 that modelling choices alone can shift recent results by 20% or more. As of September 2026, METR says the page is no longer actively updated.
• 50% is not reliable. At an 80% success rate, the horizons are much shorter: about 70 minutes for Claude Opus 4.6 and about 3 hours for the Mythos preview. For real work, you usually need the 80% or 99% number.
• The tasks are mostly software. They’re cleaner and better specified than most real jobs.

Agents in the real world

The reasoning bet’s practical payoff is agents: models that use a computer, write and run code, and keep going for many steps.

On October 22, 2024, Anthropic released “computer use”, letting Claude operate a computer by looking at screenshots and moving a mouse and keyboard. On OSWorld, a benchmark of real computer tasks, it scored 14.9% with screenshots only, against 7.8% for the next-best system. The OSWorld authors measured humans at about 72%. Anthropic itself called the feature “experimental” and “at times cumbersome and error-prone.”

Progress after that was fast. In December 2025, the startup Simular reported 72.6% on OSWorld with its open Agent S3 framework, just above the human figure, using a method that runs several attempts and picks the best. In June 2026, the OSWorld team released a harder OSWorld 2.0.

But benchmarks and real productivity are different things. In July 2025, METR published a randomised trial with 16 experienced open-source developers working on 246 real issues in projects they knew well. With AI tools (mainly Cursor with Claude 3.5 and 3.7 Sonnet), they took 19% longer. Before the study, they expected AI to speed them up by 24%. Afterwards, they still believed it had sped them up by about 20%.

In February 2026, METR published a follow-up with newer tools. It concluded that developers were likely sped up more by early 2026 than in early 2025, but that its data was only weak evidence for how much, partly because many developers no longer wanted to work without AI at all, which biased who took part. Honest summary: coding agents got genuinely useful, and measuring exactly how useful is still hard.

Do reasoning models really reason?

This is the debate underneath the whole bet.

In June 2025, six Apple researchers led by Parshin Shojaee and Iman Mirzadeh published “The Illusion of Thinking”. They tested reasoning models on puzzles where difficulty can be dialled up precisely, like the Tower of Hanoi (moving a stack of discs between pegs with strict rules) and river-crossing puzzles. They found three regimes. On easy puzzles, ordinary models did as well or better. On medium ones, thinking helped. On hard ones, both collapsed to zero. Strangest of all, as puzzles got harder, the reasoning models eventually thought less, even with token budget to spare. The authors concluded the models struggle with exact step-by-step computation and don’t reliably apply algorithms. The paper was accepted at NeurIPS 2025.

Critics pushed back quickly. A response by Alex Lawsen, titled “The Illusion of the Illusion of Thinking” (its first version listed Anthropic’s Claude Opus as a co-author), argued that:

• Some Tower of Hanoi failures happened where the full list of moves would exceed the models’ output limits, and models sometimes said they were stopping to save space.
• Some river-crossing puzzles in the test had no valid solution, yet models were marked as failing them.
• When asked to write a program that generates the solution, instead of listing every move, models did well on sizes previously reported as total failures.

A third paper in July 2025, “Rethinking the Illusion of Thinking”, re-ran the tests and found a mixed picture: models did well on solvable river-crossing puzzles but still struggled with large Tower of Hanoi instances, even step by step.

What’s fair to both sides: reasoning models clearly do something more useful than pattern-matching single answers. They solve new maths problems and fix real code. But their reasoning is also brittle in ways a careful person’s isn’t, and long exact procedures remain a weak spot. Whether to call that “real reasoning” is partly a question about the word. The practical question is narrower: on which tasks can you trust it, and how would you know?

October 2026: where the bet stands

At the time of writing:

• Reasoning models are the default. Every frontier lab ships models that think before answering, with adjustable effort levels, and reinforcement learning on checkable tasks is a standard part of training.
• Puzzle and exam benchmarks are falling fast. ARC-AGI-2 went from about 4% to 95% in roughly 18 months. Humanity’s Last Exam went from under 10% to about 55%.
• Open, interactive learning is the frontier. ARC-AGI-3 tests whether an agent can figure out an unfamiliar environment by itself. Under ARC’s standard setup, the best score rose from 0.51% at launch (March 2026) to 62.7% (September 2026), with humans at 100% and a big unanswered question about harnesses.
• Agents can work for hours, not reliably for days. METR’s 50% horizons passed 12 hours in 2026, but 80% horizons are a fraction of that, and METR’s own measuring tool has hit its limit.
• Real-world productivity gains are real but hard to pin down.

Did teaching models to think get us to AGI? It got further, faster, than most people expected in 2024. Chollet’s line from that December day still works as a summary of the field: big step, not the destination.

Scoreboard

Key ideas

• Chain of thought showed that letting a model write out its steps unlocks reasoning it already had. “Let’s think step by step” quadrupled some scores.
• o1 (2024) turned this into training, with a new scaling law: more reinforcement learning and more thinking time both improve results.
• DeepSeek-R1 (2025) showed openly that rewarding correct answers alone can teach a model to reason at length.
• ARC-AGI tracks the bet best: ARC-AGI-1 and ARC-AGI-2 fell, ARC-AGI-3 (interactive games) is the current test, and harness choice can swing results enormously.
• METR’s time horizon has doubled roughly every 4 to 7 months, but 50% success is not reliability, and the measurement has hit its limit.
• Whether models “really” reason is still debated. Their reasoning is genuinely useful and still brittle on long exact procedures.

Sources: ARC Prize, “OpenAI o3 Breakthrough High Score on ARC-AGI-Pub” (December 2024, with later cost updates), “ARC-AGI-2 + ARC Prize 2025 is Live!” (March 2025), “ARC Prize 2025 Results & Analysis” (December 2025), “ARC-AGI-3 launch” (March 2026), “OpenAI’s GPT-6 Astra on ARC-AGI-3” (September 2026), leaderboard (data as of October 19, 2026), GPT-6 Astra results and Kimi K3 results; Wei et al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” (2022); Kojima et al., “Large Language Models are Zero-Shot Reasoners” (2022); OpenAI, “Learning to Reason with LLMs” (September 2024); DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” (January 2025; Nature, 2025); Bloomberg, “Nvidia’s $589 Billion DeepSeek Rout Is Largest in Market History” (January 2025); CO/AI on the Nature paper’s cost figures (September 2025); Phan et al., “Humanity’s Last Exam” (2025) and Scale AI leaderboard (checked October 2026); METR, “Measuring AI Ability to Complete Long Tasks” (March 2025), time horizons page and data (last updated September 2026), “Impact of modelling assumptions on time horizon results” (March 2026), developer productivity study (July 2025) and February 2026 update; Anthropic, “Introducing computer use” (October 2024); Xie et al., OSWorld; Simular, “Simular’s computer-use agent outperforms humans” (December 2025); Shojaee et al., “The Illusion of Thinking” (2025); Lawsen, “The Illusion of the Illusion of Thinking” (2025); Dellibarda Varela et al., “Rethinking the Illusion of Thinking” (2025).

Related: Bet #4: Just Make It Bigger (Scale), The Other Bets: Neurosymbolic, World Models, Brains and Evolution, and What an AI Agent Actually Is.