Why AI Runs on GPUs

In 2012, three researchers at the University of Toronto entered an image recognition contest called ImageNet. Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton had built a neural network with 60 million numbers inside it, and they needed to train it on 1.2 million labelled photos.

They didn’t use a supercomputer. They used two NVIDIA GeForce GTX 580 cards, the kind of graphics card people bought to play video games. Each had 3 GB of memory. Training took five to six days.

Their network, later called AlexNet, won. Its top-5 error rate in the 2012 contest was 15.3%. The second-best entry got 26.2%. (Top-5 error means the right answer wasn’t in the model’s five best guesses.)

Two small details from their paper explain this whole post. First, the network was too big for one card, so they split it across two, with half the network on each and the cards only talking at certain layers. Second, they wrote that the size of their network was limited mainly by GPU memory and by how long they were willing to wait. They added that their results could probably be improved just by waiting for faster GPUs.

They were right. Fourteen years later, AI still runs on GPUs, the networks are still limited by memory, and the biggest models are split across tens of thousands of chips instead of two. This post explains why.

If tokens, attention and next-token prediction are new to you, start with the basics first. Here I assume you know roughly what a language model does and want to know what it runs on.

The problem: AI is a mountain of simple arithmetic

Strip away the jargon and a neural network does one kind of work, over and over: multiply numbers and add them up.

Every layer of a model takes a list of numbers (a vector) and multiplies it by a big grid of learned numbers (a matrix, the model’s “weights”). The result is a new list of numbers, which goes into the next layer. Attention, feed-forward blocks, the final step that scores every possible next token: under the hood, nearly all of it is matrix multiplication.

A useful rule of thumb from OpenAI’s 2020 scaling laws paper: running a model forward costs about 2 operations per parameter per token. One multiply, one add. Training costs about 6 per parameter per token, because the model also has to work backwards to figure out how to adjust each number.

So for a model with 8 billion parameters, producing a single token takes roughly 16 billion operations. Training a model with 405 billion parameters on 15.6 trillion tokens, like Meta’s Llama 3 405B, comes out at about 3.8 × 10²⁵ operations. That’s the figure Meta reports, and the 6-per-parameter rule lands right on it.

Two things about this work matter for hardware:

Each operation is trivial. Multiply two numbers, add to a running total. No decisions, no branching, no “if this, then that”.
Most of the operations don’t depend on each other. The thousands of multiplications inside one matrix multiply can all happen at the same time.

That second point is the whole story. If the work can be done in parallel, you want a chip that does many simple things at once rather than one complicated thing very fast.

CPU vs GPU: a few smart cores vs thousands of simple ones

NVIDIA’s own CUDA programming guide puts the difference simply. A CPU is designed to run one sequence of operations (a “thread”) as fast as possible, and runs a few tens of threads in parallel. A GPU is designed to run thousands of threads in parallel, and it spends more of its chip on doing arithmetic and less on caching and flow control.

An analogy: a CPU is a small team of senior engineers. Each one can handle anything you throw at them, jump between tasks, make judgment calls. A GPU is a stadium full of people with calculators, each doing one sum when told. If your job is “add up these 10 million pairs of numbers”, the stadium wins easily. If your job is “run this web server and handle whatever comes in”, the senior engineers win.

Where the analogy breaks: GPU cores aren’t independent people. They work in groups that all perform the same instruction at the same time on different data. That’s exactly what matrix maths needs, and exactly what most everyday software doesn’t.

Some numbers to make it concrete (at the time of writing, October 2026):

AMD’s top server CPU from its EPYC 9005 line, launched in October 2024, has 192 cores.
NVIDIA’s H100 GPU (SXM version, from 2022) has 16,896 of the basic arithmetic cores NVIDIA calls CUDA cores, plus 528 “tensor cores”, which I’ll come back to.
The GTX 580 that AlexNet used had 512 CUDA cores.

A CPU spends its chip on a few cores that handle anything; a GPU spends it on thousands of cores that all do the same simple sum.

How a gaming chip ended up running AI

GPUs were built for games. Drawing a frame means working out the colour of millions of pixels, and each pixel’s maths is mostly independent of the others. That’s the same shape of problem as a matrix multiply.

In November 2006, NVIDIA introduced CUDA, a way to program its graphics cards for general maths instead of only graphics. Researchers could now write their own code for these chips. AlexNet’s team wrote a fast GPU version of the core operation in their network, and that’s a big part of why they could train something so large in under a week.

The real bottleneck: getting numbers to the chip

Here’s the part most people miss. The hard part isn’t doing the arithmetic. It’s moving the numbers.

Every weight in a model has to travel from the GPU’s memory into its cores before it can be used. How fast that can happen is called memory bandwidth, measured in bytes per second.

Let’s do the arithmetic for a single user chatting with an 8-billion-parameter model on one H100. (These are rough upper bounds to show the shape of the problem, not benchmarks.)

The weights. At 16-bit precision, each parameter takes 2 bytes. 8 billion × 2 = 16 GB of weights.
Memory time. To produce one token, the GPU has to read all 16 GB. The H100’s memory moves 3.35 TB per second. 16 GB ÷ 3.35 TB/s ≈ 4.8 milliseconds.
Compute time. The token needs about 16 billion operations. NVIDIA lists the H100 at 1,979 trillion 16-bit operations per second “with sparsity”, a special case; the normal (dense) figure is half that, about 989 trillion. 16 billion ÷ 989 trillion ≈ 0.016 milliseconds.

For one user, the GPU spends roughly 300 times longer fetching weights than doing maths on them.

The chip could do the maths about 300 times faster than memory can feed it. In that situation, people say the work is memory-bound. The rough ceiling for one user is about 1 ÷ 4.8 ms ≈ 200 tokens per second, no matter how many cores the GPU has.

This is why providers batch users together. Read the weights once, then use them for 32 or 64 different conversations at the same time. The memory cost is shared, and the idle arithmetic units finally get work. It’s also why the KV cache from part 3 matters so much: it’s more data that has to be read for every token.

The gap keeps growing

A 2024 paper from UC Berkeley researchers, “AI and Memory Wall”, measured how this has changed over 20 years. Peak server compute grew about 3.0× every two years. Memory bandwidth grew about 1.6×, and the bandwidth of links between chips about 1.4×. Over 20 years, that’s roughly 60,000× more compute against 100× more memory bandwidth.

You can see it in the hardware. By my arithmetic (cores × clock × 2), the GTX 580 could do about 1.6 trillion 32-bit operations per second, and its memory moved 192.4 GB per second. The H100’s dense 16-bit tensor-core figure is roughly 600 times higher. Its memory bandwidth is only about 17 times higher. (Not a perfect comparison, since the number formats differ, but the direction is clear.)

Compute got cheap. Moving data didn’t. A lot of modern AI engineering is about working around that.

Tensor cores and smaller numbers

Two of the biggest workarounds are special hardware and smaller numbers.

Tensor cores

In 2017, NVIDIA’s Volta generation (the V100) added tensor cores: units built to do one thing, a small matrix multiply-and-add, in a single step. On the V100, each tensor core multiplied two 4×4 grids of 16-bit numbers and added the result to a third grid, every clock cycle. A normal core does one multiply-add at a time. A tensor core does 64.

NVIDIA’s data-centre GPUs have kept and expanded them ever since, and they’re where the headline numbers come from. The H100 does about 67 trillion operations per second in standard 32-bit maths, and about 989 trillion (dense) in 16-bit on its tensor cores.

Lower precision

The second trick is to use fewer bits per number.

32-bit (“full precision”): 4 bytes per number. The old default.
16-bit (FP16 or BF16): 2 bytes. A 2017 paper from Baidu and NVIDIA showed you could train networks mostly in 16-bit without losing accuracy, by keeping a 32-bit master copy of the weights and scaling small values so they didn’t round to zero. That “mixed precision” training is now standard.
8-bit (FP8): 1 byte. The H100 added hardware support. NVIDIA lists it at twice the 16-bit speed.
4-bit (FP4): half a byte. NVIDIA’s Blackwell generation supports it, mainly for running models rather than training them.

Halving the bits does two things at once. The tensor cores can do roughly twice as many operations per second, and every byte of memory bandwidth carries twice as many numbers. Since memory is the bottleneck, that second effect often matters more.

The catch is accuracy. Fewer bits means coarser numbers, and some parts of a model are more sensitive than others. That’s the whole topic of part 4 on quantization.

Why training needs thousands of GPUs

Let’s go back to Llama 3 405B, because Meta published unusually detailed numbers.

The compute. Meta trained it with about 3.8 × 10²⁵ operations. They report getting 38-43% of the H100’s peak 16-bit speed in practice (that ratio is called model FLOPs utilisation). Call it 400 trillion useful operations per second per GPU. On one GPU, that’s about 9.5 × 10¹⁰ seconds, or roughly 3,000 years. On 16,384 GPUs, it’s around two months. That’s my arithmetic, not Meta’s figure, but it shows why nobody trains a frontier model on one machine.

The memory. The weights alone, at 2 bytes each, are 810 GB. One H100 has 80 GB. Training also needs the gradients and the optimiser’s running state for every parameter, which adds several times more. The model simply doesn’t fit on one chip, just as AlexNet didn’t fit on one GTX 580.

So the model and the work get split up, in several ways at once: different GPUs handle different chunks of data, different layers, and different slices of each matrix. Meta used up to 16,000 H100s for Llama 3 405B.

The network becomes the computer

Once the work is split, the GPUs have to constantly share results. After every training step, every GPU’s updates have to be combined. If that sharing is slow, thousands of expensive chips sit waiting.

That’s why AI clusters are built in layers of speed:

Inside a server, 8 H100s are joined by NVLink, NVIDIA’s direct GPU-to-GPU link, at 900 GB/s per GPU.
Between servers, Meta’s Llama 3 cluster used 400 Gbps network links between GPUs. 400 gigabits is 50 gigabytes per second, so roughly 18 times slower than NVLink.
Newer systems stretch the fast zone. NVIDIA’s GB200 NVL72 rack puts 72 Blackwell GPUs in one NVLink domain at 1.8 TB/s per GPU.

Engineers arrange the split so that the chattiest work stays inside the fastest zone, and only the less frequent sharing crosses the slower network.

At this scale, things break

The Llama 3 paper has a refreshingly honest section on failures. In a 54-day stretch of training, the job was interrupted 466 times. 419 of those were unexpected, and about 78% of those were confirmed or suspected hardware problems. Faulty GPUs were the single biggest cause. Even so, Meta reports more than 90% effective training time, because they built automation to detect failures, restart from saved checkpoints and carry on.

When you have 16,000 of anything, something is always broken. A big training run is as much an operations problem as a maths problem.

TPUs and other accelerators

GPUs aren’t the only option. They’re general-purpose parallel chips that happen to be great at AI. Some companies build chips for AI only.

The best known is Google’s TPU (Tensor Processing Unit). Google’s 2017 paper described the first one, in its data centres since 2015 and built only for running models. Its core was a grid of 65,536 8-bit multiply-add units. Data flows through the grid in waves, each unit passing partial results to its neighbour, so numbers are reused many times for each trip to memory. On Google’s own workloads, it was on average 15 to 30 times faster than the GPU and CPU it was compared with (an NVIDIA K80 and an Intel Haswell server CPU, both from that era), and 30 to 80 times better in performance per watt.

Google has kept going. Its seventh-generation TPU, Ironwood, announced in April 2025, is listed in Google Cloud’s documentation with 192 GiB of memory per chip, about 7.4 TB/s of memory bandwidth, and pods of up to 9,216 chips.

Others include Amazon’s Trainium chips, which AWS sells in its cloud for training and running models, and a range of startups with their own designs. The trade-off is always the same: a chip built for one job can be faster and more efficient at it, but you give up flexibility and the huge software ecosystem around NVIDIA’s CUDA. That software lead is a big reason GPUs still dominate.

Notice what Google’s 2017 paper also said: putting the GPU’s faster memory into the TPU would have tripled its speed. Even a chip designed from scratch for AI hit the memory wall.

Training vs inference: two very different bills

Training is building the model. It happens once per model (plus experiments), on a huge cluster, for weeks or months. It’s mostly compute-bound, because big batches of training data keep the arithmetic units busy.

Inference is using the model: every question anyone asks it, for as long as it’s in service. Each request is small, but there are millions of them, every day. For a single user it’s memory-bound, as we saw, and providers spend a lot of effort batching requests to use their hardware well.

Which costs more overall? It depends on how popular the model is. One public data point: Google reported in 2022 that machine learning was 10-15% of its total energy use over the previous three years, split roughly three-fifths inference and two-fifths training. A big training run is a dramatic one-off bill. Inference is the meter that never stops running.

Limits and open questions

The memory wall isn’t solved. New chips add more and faster memory (stacked “HBM” memory next to the chip), but compute keeps growing faster. Expect continued work on smaller numbers, smarter caching and models that read fewer weights per token, like the mixture-of-experts models from part 2.
Specs are marketing. Headline figures are often “with sparsity” (a special case that doubles the number) or in the smallest number format the chip supports. Real training runs hit 40% of peak, and that’s considered good.
Power and cooling are now the limit for many data centres. A single H100 is rated up to 700 W. Multiply by tens of thousands.
The software moat is real but not permanent. Google’s Gemini report says its models were trained on TPUv4 and TPUv5e chips, which shows there are real alternatives at the very top end.

What it means in practice

If you’re building with AI or buying it, a few things follow from all this:

Token prices reflect memory, not just maths. Long prompts and long conversations cost more because there’s more to read for every token. Keep context tight when you can.
Bigger models are slower per user, not just pricier. Every token needs a full read of the active weights. If speed matters, a smaller or quantized model often beats a bigger one.
Work that can wait is cheaper. Anthropic, for example, charges 50% less for requests sent through its batch API, which returns results within hours instead of seconds (at the time of writing, October 2026). Packing requests together is exactly what the hardware likes.
Running models yourself is mostly a memory question. Before asking “how fast is this GPU?”, ask “does the model fit in its memory, and how fast can it read it?”
Be sceptical of headline chip numbers. Ask what precision, whether sparsity is included, and what real-world utilisation looks like.

Key ideas

Neural networks are mostly matrix multiplication: billions of tiny, independent multiply-and-add steps.
A CPU has a few powerful cores; a GPU has thousands of simple ones that all do the same sum at once, which fits AI perfectly.
The real bottleneck is usually memory bandwidth: moving weights to the cores takes far longer than the maths.
Tensor cores and lower precision (16-bit, 8-bit, 4-bit) squeeze more work out of every byte moved.
Frontier models need thousands of GPUs because neither the compute nor the weights fit on one chip, and fast links between chips decide how well they work together.
TPUs and other custom chips trade flexibility for efficiency, and still hit the same memory wall.
Training is a huge one-time bill; inference is a meter that runs for as long as people use the model.

Sources: Krizhevsky, Sutskever and Hinton, “ImageNet Classification with Deep Convolutional Neural Networks” (NeurIPS 2012); NVIDIA, CUDA C++ Programming Guide, sections 1.1-1.2; GTX 580 specifications via TechSpot; Kaplan et al., “Scaling Laws for Neural Language Models” (2020); Llama Team, Meta, “The Llama 3 Herd of Models” (2024); NVIDIA, H100 product page and GB200 NVL72 page, checked October 2026; AMD EPYC 9005 launch (October 2024); Gholami et al., “AI and Memory Wall” (IEEE Micro, 2024); Volta V100 announcement coverage, Hexus (May 2017), and Markidis et al., “NVIDIA Tensor Core Programmability, Performance & Precision” (2018); Micikevicius et al., “Mixed Precision Training” (ICLR 2018); Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit” (ISCA 2017); Google Cloud, TPU7x (Ironwood) documentation, checked October 2026; AWS Trainium; Gemini Team, Google, “Gemini: A Family of Highly Capable Multimodal Models” (2023); Anthropic, batch processing docs, checked October 2026; Patterson, “Good News About the Carbon Footprint of Machine Learning Training” (Google Research, February 2022). Per-token and training-time estimates are my own arithmetic from these figures.