How AI Works
19 essays on this topic.
Why Do Neural Networks Generalize? Overfitting, Double Descent and Grokking
Big networks memorise their data and still work on new data. Overfitting, double descent and grokking in plain English, with a numpy demo you can run.
The Math Behind AI, Learned From What You Already Build
Six areas of math explain almost everything in modern AI. Plain English, one runnable numpy example each, and a 12-week path for developers.
Inside a Transformer: One Word's Journey to the Next Word
Follow one word through a language model, from token ID to the final pick: embeddings, a stack of layers, attention, feed-forward memory and probabilities.
How AI Knows What "It" Means: Attention Explained
How does AI know what "it" refers to? A plain-English guide to attention: queries, keys and values, softmax, heads, masking and why long context costs more.
How AI Turns Words Into Numbers: Embeddings Explained
A token ID is just a label. An embedding is a learned list of numbers that puts each word on a map of meaning. Here's how it works, and where it shows up.
What's a Token, and Why AI Has Limits
AI doesn't read letters or words. It reads tokens. A plain-English guide to tokens, token IDs, context windows, per-token pricing and the strawberry problem.
How ChatGPT Actually Works, Explained Simply
ChatGPT, Claude and Gemini predict the next word, one piece at a time. A plain-English guide to how they learn, why they sound confident when wrong, and what they are not.
World Models: Can AI Simulate Reality?
What a world model is, the evidence that AI builds one inside, the LeCun debate, and what Sora and Genie can and can't simulate yet.
Synthetic Data and Model Collapse
What happens when AI trains on AI-written text? Model collapse explained, why mixing in real data prevents it, and how labs use synthetic data well.
AI Benchmarks and Goodhart's Law: Why Leaderboards Mislead
Why AI leaderboard scores drift away from real quality: Goodhart's law, saturation, contamination, arena gaming, and how to build your own small test set.
Reward Hacking: When AI Wins the Wrong Game
From a boat circling a lagoon to coding agents that edit the tests: why AI learns to game its reward, how labs catch it, and what it means for your metrics.
Glitch Tokens: The Strange Words That Broke GPT-3
Why ' SolidGoldMagikarp' broke GPT-3: how tokenizers and models are trained apart, under-trained tokens, how labs fixed it, and other tokenizer quirks.
How Image Generators Work: Diffusion Explained
How AI image generators turn random noise into pictures: denoising, text encoders, latent diffusion, guidance, why hands failed, and the copyright fights.
How AI Sees: Vision Transformers and CLIP
Why a paper note saying "iPod" fooled an AI looking at an apple, and how vision transformers, CLIP and chat models really turn pictures into tokens.
Why AI Runs on GPUs
From two gaming cards in 2012 to clusters of 16,000 chips: why AI runs on GPUs, why memory speed matters more than raw power, and what it costs.
Shrinking AI: Quantization and Distillation Explained
How big AI models get small enough for a laptop or phone: quantization, distillation, pruning, what each one costs, and the licence questions.
Long Context and the KV Cache: How AI Reads a Whole Book at Once
How AI models read hundreds of thousands of tokens at once: why attention gets expensive, what the KV cache is, and when long context beats RAG.
Mixture of Experts: Why Giant Models Only Use Part of Their Brain
How mixture-of-experts models work: routers, top-k experts, load balancing, what experts really learn, and why total and active parameters differ.
Interpretability: How Scientists Look Inside an AI's Mind
Superposition, sparse autoencoders, Golden Gate Claude and attribution graphs: how researchers find concepts inside a model, and what they still can't see.
