Why Do Neural Networks Generalize? Overfitting, Double Descent and Grokking

A small neural network is learning to divide numbers. Not ordinary division: division modulo 97, the kind of arithmetic where numbers wrap around like hours on a clock that has 97 hours. Every question has the form a ÷ b = ?, and every answer is one of 97 symbols.

There are only so many questions, so the researchers wrote out the whole table and gave the network half of it to learn from. The other half was hidden. That hidden half is the validation set: questions the model never trains on, used only to check whether it learned the rule or just the answers.

Within about a thousand training steps, the network got close to 100% on the half it had seen. On the hidden half it was no better than guessing. That’s textbook overfitting: the model had memorised its homework.

Any sensible engineer stops here. The researchers kept going. For tens of thousands of steps, nothing changed. Then, somewhere past 100,000 steps, accuracy on the hidden half started to climb, and by close to a million steps it reached near-perfect. The network had gone from memorising the table to knowing how to divide.

The paper that reported this, by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra in 2022, named it grokking. Two more details matter. The network never saw numbers as numbers: each of the 97 values was just a separate symbol, so the structure had to be discovered. And of all the training tricks the authors tried, weight decay helped most, more than halving the data needed to generalize compared with most other interventions. Weight decay is a small pull, applied at every step, that shrinks every weight toward zero.

Grokking is a good door into a bigger question that still isn’t fully answered: why do neural networks work on data they’ve never seen? By the old rules, networks with more knobs than training examples should memorise and fail. Mostly, they don’t.

This post covers the old rules, the experiments that broke them, the leading explanations, and a 30-line numpy demo you can run on a laptop.

The classic picture: memorise or learn

Training a model means adjusting its parameters (the learned numbers inside it, also called weights) so its answers on the training set get better. The number we watch is the error: how wrong the model is on average.

But you already know the training answers. What matters is the error on new examples from the same kind of data, the test error. A model generalizes when its test error is close to its training error.

The textbook story about when that happens is the bias-variance trade-off.

• Bias is error from a model that’s too simple. A straight line can’t follow a curve. This is underfitting: high training error and high test error.
• Variance is error from a model that’s too flexible. It bends to fit every quirk and bit of noise in its training set. This is overfitting: low training error, high test error.

Make the model bigger and bias falls while variance rises. Add them up and test error traces a U-shaped curve: high on the left (too simple), lowest at a sweet spot, high again on the right (too complex). The textbook advice follows directly: pick the size at the bottom of the U, and if a model fits its training data perfectly, be suspicious.

Regularisation means anything that stops a model using all its flexibility: weight decay, dropout (randomly switching off parts of the network during training), data augmentation (training on flipped, cropped or noisy copies of each example), or early stopping (stopping training when validation error starts to rise).

This picture works well for many classic methods. The trouble started with deep learning.

The experiment that broke it

In 2017, Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht and Oriol Vinyals published “Understanding deep learning requires rethinking generalization” at ICLR. The core experiment fits in two sentences.

Take a standard image dataset (they used CIFAR10 and ImageNet). Replace every label with a random one, so a picture of a dog might be labelled “truck”, and train a standard image classifier on it.

There is nothing to learn. The labels have no relationship to the pictures. Yet on CIFAR10 the networks reached zero training error, memorising 50,000 meaningless picture-label pairs, and training took only a small constant factor longer than on the real labels. Test error, of course, was no better than chance. The authors summed it up in one line: “Deep neural networks easily fit random labels.”

They even replaced the images with pure random noise; the networks still fit them.

Here’s why that matters. The same network, same size, same settings, can learn real labels and generalize well, or memorise random labels and generalize not at all. So “too big” can’t explain when overfitting happens. Classic capacity measures, which look only at the model family, can’t tell the two cases apart.

Their second finding hit regularisation. With weight decay, dropout and data augmentation switched off, the networks often still generalized well on real labels. The paper’s verdict: explicit regularisation may improve test performance, but is neither necessary nor by itself sufficient to control generalization error.

So a big network can memorise anything, and usually doesn’t. The rest of this post is about why.

Double descent: the U-curve has a second half

In 2019, Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal published “Reconciling modern machine-learning practice and the classical bias-variance trade-off” in PNAS. They plotted test error against model size, but kept going past the point where the U-curve usually stops.

The landmark is the interpolation threshold. To interpolate means to fit the training data exactly. The threshold is the smallest model size that can. For their simplest models, that’s roughly when the number of parameters equals the number of training examples.

What they found:

1. Below the threshold, test error follows the classic U.
2. At the threshold, test error spikes. The model can just barely hit every training point, so it has one way to do it, usually a wild, contorted fit.
3. Past the threshold, test error falls again, often below the bottom of the U.

They called this shape double descent: test error goes down, up, and down again.

Their main evidence included random Fourier features on a subset of MNIST, the handwritten-digit dataset. A random-features model is a two-layer network whose first layer is random and frozen; only the last layer is learned. Test error peaked where the number of features equalled the number of training examples, then fell as features grew. When many settings of the last layer fit the data exactly, they picked the one with the smallest weights, and argued that this is the key: a bigger model has more exact fits to choose from, so the smallest, smoothest one gets smoother.

That’s the core idea of double descent: past the threshold, the question isn’t whether the model can fit the data, but which of its many perfect fits it ends up with.

Deep double descent

Later in 2019, Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak and Ilya Sutskever showed the same shape in real deep networks, in a paper called “Deep Double Descent: Where Bigger Models and More Data Hurt”. They found it along three different axes:

To tie them together, they defined effective model complexity: the most training examples a training procedure can fit to near-zero error. Bigger models and longer training raise it. Test error spikes when it’s close to the training set size.

Two lessons: near the threshold, more data can move the peak onto you, and early stopping helps mainly near that peak.

Nakkiran et al. saw every form of double descent most strongly with label noise, meaning some training labels are wrong, though peaks also appeared without it. The demo below adds noise for that reason.

So why does it work? Five partial answers

Nobody has one accepted explanation. Treat these as hypotheses, not settled facts.

1. The training algorithm picks simple solutions

A big network has many settings that fit the training data perfectly. This is implicit regularisation: regularisation that comes from the training method itself, not from a term you added.

The cleanest result is for linear models. Zhang et al. pointed out that gradient descent on a linear least-squares problem, started from zero, converges to the minimum-norm solution: the exact fit with the smallest weights. Soudry and colleagues (JMLR, 2018) showed that for logistic regression on data that can be perfectly separated, gradient descent slowly converges toward the maximum-margin solution: the boundary with the most room on both sides. Nobody asked for either; the algorithm brought them along.

Whether this carries over to deep networks is open, but it’s a plausible reason why “can memorise” doesn’t mean “will memorise”.

2. Flat minima versus sharp minima

Picture the loss, the number training pushes down, as a landscape. A minimum is a low point. A sharp minimum is a narrow pit: move the weights a little and the loss shoots up. A flat minimum is a wide valley.

In 2017, Nitish Shirish Keskar and colleagues (ICLR) reported that training with large batches (many examples per update) tends to land in sharp minima and generalizes worse, while small batches, whose noisy gradients keep bouncing the weights around, tend to find flat ones.

It’s debated. The same year, Laurent Dinh, Razvan Pascanu, Samy Bengio and Yoshua Bengio (ICML 2017) showed that for ReLU networks you can rescale the weights so the network computes exactly the same function, yet sits in a much sharper minimum. So sharpness, as usually measured, can’t be the whole story. On the other side, Jiang and colleagues (ICLR 2020) compared dozens of proposed generalization measures across thousands of trained networks and found that sharpness-based measures were among the best predictors. Flatness seems to matter, but measuring it properly is tricky.

3. Simplicity bias

Guillermo Valle-Pérez, Chico Camargo and Ard Louis (ICLR 2019) argued that if you pick network weights at random, the function you get is far more likely to be simple than complex. Real-world rules tend to be simple too, so a network that leans toward simple functions will tend to guess right on new examples.

4. What grokking looks like inside

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith and Jacob Steinhardt (ICLR 2023) trained a one-layer transformer on addition modulo 113 using 30% of all pairs, and then reverse-engineered what it had learned. The network had learned to add by turning numbers into rotations around a circle, using sine and cosine waves. (Power et al. had seen a hint: their learned symbols for modular addition formed a circle.)

They also built progress measures: numbers that track how much of the final algorithm exists inside the network during training. Those measures showed that grokking isn’t sudden on the inside. Training split into three phases:

1. Memorisation: the network fits the training pairs as a lookup table.
2. Circuit formation: the general algorithm slowly builds up alongside the memorised table, while validation accuracy still looks flat.
3. Cleanup: weight decay removes the now-unneeded memorised parts, and validation accuracy finally jumps.

Their networks did not grok without weight decay or some other regularisation. In their account, the pull toward small weights is what lets the general algorithm win over the lookup table.

So the jump you see from outside is the end of a long, gradual change you can’t see from the accuracy curve.

5. What’s still open

• No theory yet predicts, for a given real network and dataset, how well it will generalize before you test it.
• Why simple, general solutions win in large language models trained on messy data is far less clear than in a modular arithmetic toy.
• Power et al. note these effects are far more pronounced on small algorithmic datasets than on natural data.

Run it yourself: double descent on a laptop

Here’s the classic demo, the same idea as Belkin et al.’s random-features experiment, in plain numpy. It doesn’t reproduce grokking; it shows overfitting and double descent as the model grows.

• 100 training examples, each a list of 10 random numbers. The target is a smooth rule plus some noise.
• A random-features model: a fixed random layer followed by ReLU (keep positives, zero out negatives). Only the last layer is fitted, with least squares.
• We vary the number of features p. When p is bigger than 100, there are many exact fits; np.linalg.pinv (the pseudo-inverse) picks the one with the smallest weights.

import numpy as np

rng = np.random.default_rng(0)
n_train, n_test, d = 100, 1000, 10

# A smooth hidden rule, plus noise on the training labels
w_true = rng.normal(size=d) / np.sqrt(d)
def make_data(n):
    X = rng.normal(size=(n, d))
    y = np.tanh(X @ w_true) + 0.3 * rng.normal(size=n)
    return X, y
X_train, y_train = make_data(n_train)
X_test, y_test = make_data(n_test)

# Random features: a fixed random layer + ReLU. Only the last layer is fitted.
W = rng.normal(size=(d, 2000)) / np.sqrt(d)
def features(X, p):
    return np.maximum(0, X @ W[:, :p])

print(f"{'features':>8} {'train MSE':>10} {'test MSE':>9}")
for p in [5, 20, 50, 80, 95, 100, 105, 120, 200, 500, 2000]:
    F_train, F_test = features(X_train, p), features(X_test, p)
    # least squares; when p > n there are many exact fits, and
    # pinv picks the one with the smallest weights (minimum norm)
    a = np.linalg.pinv(F_train) @ y_train
    train = np.mean((F_train @ a - y_train) ** 2)
    test = np.mean((F_test @ a - y_test) ** 2)
    print(f"{p:>8} {train:>10.3f} {test:>9.3f}")

Output (numpy 2.5.4, seed 0):

features  train MSE  test MSE
       5      0.242     0.332
      20      0.129     0.182
      50      0.072     0.248
      80      0.019     1.230
      95      0.004     8.380
     100      0.000   234.907
     105      0.000     3.972
     120      0.000     0.858
     200      0.000     0.419
     500      0.000     0.348
    2000      0.000     0.302

MSE is mean squared error: the average squared gap between prediction and truth. Reading down:

• 5 to 20 features: both errors fall. The classic left side of the U.
• 50 to 95 features: training error keeps falling, but test error rises. Classic overfitting.
• 100 features: training error hits zero, and test error explodes to 234.9. That’s the interpolation threshold: exactly as many features as examples, one exact fit, and it’s wild.
• Beyond 100: training error stays at zero, yet test error falls again, down to 0.302 at 2,000 features.

Two notes. Here, the second descent doesn’t quite beat the sweet spot at 20 features (0.182), so “bigger is better” isn’t automatic. And I re-ran it with seeds 1, 2 and 3: the numbers change, but the peak landed at 100 features every time.

Two things to try, which I also ran: set the noise 0.3 to 0.0 and the peak drops from 234.9 to 8.9 (and 2,000 features now beat every smaller model); set n_train to 200 and the peak moves to 200 features.

Practical rules for your own models

More data or a bigger model? If training error is high, you’re underfitting: try a bigger model. If training error is low and test error falls as data grows, get more data.

Leakage means information from the test set sneaking into training. It’s a common reason a model looks brilliant offline and fails in production, and one way benchmarks stop measuring what they claim.

What it means for RIAI

RIAI is my research project on agents that learn from their own experience. Two of its results read differently after this post. These are questions, not answers.

Why can’t 11 examples move a model? In the JEPA post, the agent’s own verified solutions were traced, added to training, and the model was fine-tuned for 3 minutes. It imagined 2 of 11 of its own solutions exactly right before, and still 2 of 11 after. As the research review notes, those 11 traces joined a corpus built from 61,611 traced programs. On a learning curve, 11 extra examples is a point so far left that no movement is what you’d expect. The better question: what does the curve look like with 100, 1,000 and 10,000 self-generated examples? Only a curve can tell a failed method from a dose that was too small.

Is “2 of 11” even a measurement? An all-or-nothing score on 11 items can’t show partial progress. Grokking makes the same point from the other side: accuracy can sit flat while the useful structure grows underneath. Would a continuous measure, like the rank of the correct program or the loss on held-out traces, have moved when exact match didn’t?

Memorisation or understanding? The JEPA planner, run closed loop (re-reading the real program state after every step), matched 91% of 167 held-out random programs on its first plan. Those programs were built from the same blocks but never seen as whole programs in training. That’s evidence of more than a lookup table, but every block was seen. Open questions:

• How does that 91% change on programs that are longer than anything in training?
• Do the matched programs still behave correctly on fresh inputs, not just the examples used to search?
• Do any held-out programs have near-duplicates in the training data? A leakage check would make the number much stronger.

Key ideas

• Grokking: a network can memorise its training data and only generalize many steps later. Weight decay helped most.
• The U-curve is real but incomplete: it covers models too small to fit their training data exactly.
• Deep networks can memorise random labels, so size alone can’t explain overfitting, and explicit regularisation isn’t the main reason they generalize.
• Double descent: test error spikes at the interpolation threshold, then falls again, by model size, training time and sometimes data size.
• The explanations are hypotheses: implicit regularisation, flat minima (debated), simplicity bias, and gradual circuits inside grokking.
• In practice: truly held-out data, learning curves, size sweeps and a leakage check beat any single score.

Sources: Power, Burda, Edwards, Babuschkin and Misra, “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets” (2022); Zhang, Bengio, Hardt, Recht and Vinyals, “Understanding deep learning requires rethinking generalization” (ICLR 2017); Belkin, Hsu, Ma and Mandal, “Reconciling modern machine-learning practice and the classical bias-variance trade-off” (PNAS 116(32), 2019; arXiv); Nakkiran, Kaplun, Bansal, Yang, Barak and Sutskever, “Deep Double Descent: Where Bigger Models and More Data Hurt” (2019); Keskar, Mudigere, Nocedal, Smelyanskiy and Tang, “On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima” (ICLR 2017); Dinh, Pascanu, Bengio and Bengio, “Sharp Minima Can Generalize For Deep Nets” (ICML 2017); Jiang, Neyshabur, Mobahi, Krishnan and Bengio, “Fantastic Generalization Measures and Where to Find Them” (ICLR 2020); Soudry, Hoffer, Nacson, Gunasekar and Srebro, “The Implicit Bias of Gradient Descent on Separable Data” (JMLR 2018); Valle-Pérez, Camargo and Louis, “Deep learning generalizes because the parameter-function map is biased towards simple functions” (ICLR 2019); Nanda, Chan, Lieberum, Smith and Steinhardt, “Progress measures for grokking via mechanistic interpretability” (ICLR 2023). Links checked November 2026. The code output was produced by running the snippet with numpy 2.5.4.

Related: The Math Behind AI, Learned From What You Already Build, AI Benchmarks and Goodhart’s Law: Why Leaderboards Mislead, and What the Research Says: Mapping RIAI Against the Field.