Bet #2: Let It Learn (Neural Networks)

On 7 July 1958, the US Navy invited reporters to watch a computer learn.

The machine was an IBM 704, a room-sized mainframe. A young Cornell psychologist named Frank Rosenblatt fed it punch cards. Some cards had a mark on the left, some on the right. Nobody had programmed a rule for telling them apart. The computer was told only when it got one wrong. After 50 tries, it had taught itself to sort them.

That is not much. A child could do it after two. But the next morning, the New York Times ran the story under the headline “New Navy Device Learns By Doing”. The article said the Navy had revealed the embryo of a computer it expected would be able to “walk, talk, see, write, reproduce itself and be conscious of its existence.”

It would be 54 years before neural networks did anything that impressive to the wider world. And when they finally did, they did it in a way Rosenblatt would have recognised.

This is the second bet in the series. Part 2 was about writing intelligence down as rules. This bet says the opposite: don’t write the rules. Build something that can learn them from examples. It is the bet that, in October 2026, every frontier AI system is built on. It is also the bet that spent the longest time being laughed at.

The idea: a brain-shaped calculator

The idea starts with a 1943 paper by Warren McCulloch, a neurophysiologist, and Walter Pitts, a self-taught logician. It was called “A Logical Calculus of the Ideas Immanent in Nervous Activity”.

They noticed that a brain cell, a neuron, behaves a bit like a switch. It collects signals from other neurons. If the total crosses some level, called a threshold, it fires. If not, it stays quiet. All or nothing.

McCulloch and Pitts showed that if you wire up enough of these simple threshold units, the network can compute any logical function: AND, OR, NOT, and anything you build from them. That was a big claim. It meant a brain-like network could, in principle, do anything a logic machine could.

But there was a catch. In their model, a human had to set every connection by hand. The network could compute, but it could not learn.

Rosenblatt’s perceptron: the first network that learned

Rosenblatt’s contribution was learning. His 1958 paper in Psychological Review described what he called a perceptron.

A perceptron is one artificial neuron with adjustable inputs. Each input has a weight, a number that says how much that input matters. The neuron multiplies each input by its weight, adds them up, and fires if the total crosses the threshold. That’s it.

The clever part is the learning rule. Show the perceptron an example. If it answers correctly, change nothing. If it’s wrong, nudge each weight a little in the direction that would have given the right answer. Repeat thousands of times. If a solution exists that a single neuron can represent, the weights will eventually find it.

Then Rosenblatt built one in hardware. The Mark I Perceptron was assembled at the Cornell Aeronautical Laboratory in Buffalo, New York, in 1959. Its “eye” was a 20 by 20 grid of 400 light sensors. Its weights were physical knobs, potentiometers, turned by small electric motors whenever the machine made a mistake. You could literally watch it learn.

The Navy said it would build a full Perceptron machine in about a year at a cost of $100,000. The press coverage promised machines that would walk, talk and see. The direction was right. The timing was off by about half a century.

The wall: Minsky and Papert’s Perceptrons (1969)

In 1969, two MIT researchers, Marvin Minsky and Seymour Papert, published a book called Perceptrons. It is one of the most cited and most misremembered books in AI history.

Here is what it actually proved.

A single-layer perceptron can only separate things that can be split by a straight line (or its higher-dimensional cousin, a flat plane). The simplest problem it cannot solve is called XOR, “exclusive or”: answer yes if exactly one of two inputs is on. Plot the four cases on a square and the yes answers sit on opposite corners. No single straight line separates them.

Minsky and Papert went much further than XOR. They proved careful theorems about perceptrons where each unit only looks at a limited patch of the input. Such networks cannot compute parity (is the number of lit pixels odd or even?) without some unit that looks at everything. And the size a perceptron needs to decide connectedness (is this shape one piece or two?) grows as the image grows.

Here is what it did not prove: that multi-layer networks were useless. Minsky and Papert knew that stacking layers could get around these limits. What they doubted was that anyone could find a way to train a multi-layer network. That was an intuition, not a theorem.

The message that reached funders was simpler: “perceptrons don’t work”. Money and attention shifted towards the symbolic approach from Part 2. Historians still argue about how much the book itself caused the slump in neural network research in the 1970s, versus other forces like overpromising and weak hardware. But the slump was real.

Rosenblatt did not live to see the comeback. He died in a boating accident on Chesapeake Bay on 11 July 1971, his 43rd birthday.

Backpropagation: how to train the hidden layers

The problem Minsky and Papert pointed at had a name: credit assignment. If a network with hidden layers (layers between input and output) gets an answer wrong, which of its thousands of weights is to blame, and by how much?

The answer is backpropagation, usually shortened to “backprop”. You run an example forward through the network and measure the error at the output. Then you use calculus (the chain rule) to work backwards, layer by layer, calculating how much each weight contributed to that error. Then you nudge every weight a little in the direction that reduces it. Do this millions of times.

The method has a tangled history:

• In 1970, Finnish student Seppo Linnainmaa described the general mathematical technique in his master’s thesis. It’s now called reverse-mode automatic differentiation. He didn’t apply it to neural networks.
• In 1974, Paul Werbos’s PhD thesis proposed using it to train neural networks. It stayed largely unknown.
• Others rediscovered it in the early and mid 1980s, including David Parker and a young Yann LeCun.
• In October 1986, David Rumelhart, Geoffrey Hinton and Ronald Williams published “Learning representations by back-propagating errors” in Nature.

The 1986 paper is the one that made it famous. Its real contribution was showing what the hidden layers learned: useful internal representations, features nobody had programmed, discovered by the network itself. Hinton has said plainly that backprop was invented independently by several people before his paper.

With backprop, the XOR problem was trivial. The 1969 wall had a door.

LeCun’s networks read the mail, then the cheques

The first time neural networks did real work in the world was in Yann LeCun’s group at AT&T Bell Labs.

In 1989, LeCun and colleagues published “Backpropagation Applied to Handwritten Zip Code Recognition”. They trained a network on digits taken from real US mail. A companion paper from the same year reported about a 1 percent error rate, with about 9 percent of digits rejected as too uncertain and handed to a person.

The key design was the convolutional network. Instead of connecting every pixel to every unit, it uses small pattern detectors (a curve, a corner, a stroke) and slides the same detector across the whole image. This builds in a useful assumption: a “7” is a “7” wherever it appears on the page. It also means far fewer weights to learn.

Then came the cheques. According to LeCun’s own account, the system went into ATMs for amount verification at a French bank, Crédit Mutuel de Bretagne, and commercial deployment in banks started in 1995. It could read about half the cheques it saw, printed or handwritten, and sent the rest to human operators. That design detail matters: the system knew when to pass.

In 1998, LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner published the long paper describing LeNet-5, the architecture that became the template for image recognition.

The long quiet years

And then, for most of the 1990s and 2000s, not much happened. At least, not much that the wider world noticed.

Neural networks worked on small problems. On bigger ones they were slow and fiddly. Deep networks, with many layers, were especially hard to train. In 1991, Sepp Hochreiter’s diploma thesis identified a key reason, now called the vanishing gradient problem: as the error signal is passed back through many layers, it shrinks until the early layers barely learn at all. In 1997, Hochreiter and Jürgen Schmidhuber published the LSTM (long short-term memory), a design that eased the problem for networks that process sequences.

Through these years, a small group kept the bet alive. In 2006, Hinton, Simon Osindero and Yee-Whye Teh published a way to train deep networks one layer at a time, then fine-tune the whole thing. It was a sign that depth could be tamed.

Two other ingredients were quietly arriving:

• Data. Fei-Fei Li and colleagues presented ImageNet at CVPR in 2009, a huge collection of labelled photos built with crowd workers. The annual ImageNet challenge used about 1.2 million training images across 1,000 categories.
• Compute. Graphics cards, built for video games, turned out to be very good at the kind of maths neural networks need. (I wrote about why in Why AI Runs on GPUs.)

2012: AlexNet

In 2012, three people from Hinton’s lab at the University of Toronto entered the ImageNet challenge: Alex Krizhevsky, Ilya Sutskever and Hinton himself.

Their network, later called AlexNet, had about 60 million weights and 650,000 neurons, in five convolutional layers and three fully connected ones. They trained it on two consumer graphics cards (Nvidia GTX 580s with 3 GB of memory each) for five to six days.

The result: a top-5 error rate of 15.3 percent. “Top-5” means the right answer wasn’t among the network’s five best guesses. The second-best entry scored 26.2 percent.

In a field where teams fought over fractions of a point, that was not a win. It was a different sport. AlexNet was not a new idea. It was LeCun’s convolutional network, trained with 1986 backprop, made big enough and fed enough data on cheap enough hardware. The bet hadn’t changed. The world had finally caught up with it.

The boom: speech, vision, translation

What followed was fast.

• Speech. In November 2012, researchers from four groups, including Hinton’s lab, Microsoft, Google and IBM, published a joint paper showing that deep networks beat the older statistical methods for speech recognition on several benchmarks, sometimes by a large margin. Voice assistants and dictation got noticeably better in the years after.
• Vision. After AlexNet, almost every serious image-recognition system became a deep convolutional network. Photo search, face unlock, medical image analysis, self-driving perception.
• Translation. In September 2016, Google switched Chinese-to-English in Google Translate to a neural system. Google reported that, in side-by-side ratings by bilingual humans, the new system cut translation errors by 55 to 85 percent on several major language pairs compared with its old phrase-based system. Google also listed what it still got wrong: dropped words, mistranslated names and rare terms.
• Games. Deep networks also became the “intuition” inside game-playing systems like AlphaGo. That’s the story of Bet #3.

And in 2017, a new network design called the Transformer arrived, which led to the large language models of today. That’s the story of Bet #4.

The prizes

The outsiders became the establishment.

On 27 March 2019, the ACM announced the 2018 Turing Award, often called the Nobel Prize of computing, for Yoshua Bengio, Geoffrey Hinton and Yann LeCun, “for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.” ACM’s press release called them the “Fathers of the Deep Learning Revolution”.

Then, on 8 October 2024, the actual Nobel. The Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to John Hopfield and Geoffrey Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.” Hopfield was recognised for the Hopfield network, a kind of memory that can store patterns and rebuild them from partial input. Hinton was recognised for the Boltzmann machine, which used ideas from statistical physics to learn the features in data.

Rosenblatt’s embryo had grown up. It just took 66 years and a physics prize to get there.

Where it hit walls

Neural networks won. But winning exposed limits that are still with us.

Data hunger

A child learns what a cat is from a handful of cats. AlexNet needed over a million labelled images to learn 1,000 categories. Every big leap in neural networks since then has come with a big leap in data. That works as long as the data keeps coming. Whether it can keep coming, and what happens when models train on their own output, is its own story (see Synthetic Data and Model Collapse).

Brittleness

Neural networks can be very good on data that looks like their training data, and strangely bad just outside it. The sharpest example is the adversarial example.

In December 2013, Christian Szegedy and colleagues showed that you could change an image in ways invisible to a human and make a network misclassify it. In a 2014 follow-up, Ian Goodfellow, Jonathon Shlens and Szegedy showed a now-famous picture. A network called GoogLeNet looked at a photo of a panda and said “panda” with 57.7 percent confidence. They added a tiny, carefully calculated layer of noise, too small for a human to notice. The network now said “gibbon” with 99.3 percent confidence.

The network wasn’t confused in a human way. It was relying on patterns in the pixels that humans don’t use at all. That gap, between what the network sees and what we see, has never fully closed.

You can’t easily read what it learned

A rule-based system from Part 2 can explain itself: here is the rule that fired. A neural network’s knowledge is spread across millions or billions of numbers. Why did it say gibbon? Scientists are making real progress on this (see Interpretability), but in 2026 it is still far from solved.

Learning, but not reasoning?

The oldest criticism, from the symbolic camp, is that neural networks learn associations, not reasoning: they match patterns rather than follow logic. Large language models have blurred that line a lot. Whether they’ve erased it is the subject of Bet #5 and the other bets on combining the two.

The legacy: everything is a neural network now

Here is the part that would have amazed Rosenblatt’s critics.

At the time of writing, October 2026, every frontier AI system I know of is a neural network trained with some version of backpropagation, running on hardware descended from the graphics cards AlexNet used. ChatGPT, Claude, Gemini, image generators, speech recognisers, translation, recommendation feeds. The symbolic bet did not disappear (it lives on in tools, search and code), but the core of today’s AI is the learning bet.

The ideas from each era are all still there:

• The weighted sum and threshold of 1943 and 1958 is still the basic unit, with smoother versions of the threshold.
• Backprop from 1970 to 1986 is still how almost everything is trained.
• Convolutions from LeCun’s cheque reader still power much of computer vision.
• Data plus GPUs from 2012 became the “scale” bet that drives the field today.

But “neural networks won” is not the same as “neural networks reach AGI”. This bet answered how a machine can learn. It did not answer what it should learn from, or what it should aim for. The next three parts are three different answers to those questions: learn from reward, learn from everything at huge scale, and learn to think step by step.

Scoreboard

Key ideas

• A neural network is a stack of weighted sums. Each unit adds up its inputs times their weights and passes the result on. Learning means adjusting the weights.
• The perceptron (1958) could learn, but only straight-line splits. Minsky and Papert proved its limits in 1969; their doubt about multi-layer networks was a conjecture, not a proof.
• Backpropagation solved credit assignment. It works out how much each weight contributed to an error, so hidden layers can learn. It was invented several times between 1970 and 1986.
• The idea was right decades before it worked. AlexNet in 2012 used old ideas plus big data and GPUs, and won by more than ten points.
• Neural networks are data-hungry, brittle in strange ways and hard to read. Adversarial examples show they can rely on patterns humans can’t see.
• In 2026 the learning bet is the foundation of everything, but it says how machines learn, not what they should learn or aim for. The next bets try to answer that.

Sources: McCulloch and Pitts, “A Logical Calculus of the Ideas Immanent in Nervous Activity” (Bulletin of Mathematical Biophysics 5:115–133, 1943); “New Navy Device Learns By Doing” (The New York Times, 8 July 1958, UPI dispatch) and Cornell Chronicle on Rosenblatt’s demonstration; Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain” (Psychological Review 65(6), 1958); Perceptron (Mark I details) and Frank Rosenblatt; Minsky and Papert, Perceptrons (MIT Press, 1969), summarised in Perceptrons (book); Schmidhuber, “Deep Learning in Neural Networks: An Overview” (2014) for the history of backpropagation; Rumelhart, Hinton and Williams, “Learning representations by back-propagating errors” (Nature 323, 1986); LeCun et al., “Backpropagation Applied to Handwritten Zip Code Recognition” (Neural Computation, 1989) and “Handwritten Digit Recognition with a Back-Propagation Network” (NIPS 1989); Yann LeCun’s account of the cheque-reading deployment (2021); LeCun, Bottou, Bengio and Haffner, “Gradient-Based Learning Applied to Document Recognition” (Proceedings of the IEEE, 1998); Hinton, Osindero and Teh, “A Fast Learning Algorithm for Deep Belief Nets” (Neural Computation, 2006); Vanishing gradient problem; Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database” (CVPR 2009); Krizhevsky, Sutskever and Hinton, “ImageNet Classification with Deep Convolutional Neural Networks” (NeurIPS 2012); Hinton et al., “Deep Neural Networks for Acoustic Modeling in Speech Recognition” (IEEE Signal Processing Magazine, 2012); Google Research, “A Neural Network for Machine Translation, at Production Scale” (September 2016); ACM, “Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award” (27 March 2019); The Nobel Prize in Physics 2024 press release (8 October 2024); Szegedy et al., “Intriguing properties of neural networks” (2013); Goodfellow, Shlens and Szegedy, “Explaining and Harnessing Adversarial Examples” (ICLR 2015).

Related: Bet #1: Write Down the Rules (Symbolic AI), Bet #3: Reward Is Enough (Reinforcement Learning), and Why AI Runs on GPUs.