# Mixture of Experts: Why Giant Models Only Use Part of Their Brain > How mixture-of-experts models work: routers, top-k experts, load balancing, what experts really learn, and why total and active parameters differ. - Author: Tai Pham Phu - Published: 20 Oct 2026 - Tags: How AI Works, LLM, Applied AI, Mixture of Experts, Model Architecture - URL: https://phamphutai.com/2026/10/20/mixture-of-experts-explained/ On Friday 8 December 2023, the French start-up Mistral AI posted a tweet with no announcement, no blog post and no picture. Just a magnet link: the kind of link you paste into a BitTorrent client. People who did that started downloading an 87 GB folder. Inside was a new model called **Mixtral 8x7B**. It wasn't even the first time. Mistral had released its previous model, Mistral 7B, the same way on 26 September. The explanation came three days later. On 11 December, Mistral published a short post. The model held **46.7 billion parameters in total, but used only 12.9 billion of them for each token**. It outperformed Meta's Llama 2 70B on most benchmarks with, Mistral said, six times faster inference, and matched or beat GPT-3.5 on most standard benchmarks. It was released under the Apache 2.0 licence, so anyone could use it. How does a model use roughly a quarter of its parameters for each token and still beat a dense model with 70 billion? The trick is in the name: **mixture of experts**, or MoE. Many of the biggest open models released since use some version of it. This post is part 2 of **How AI Works: Advanced**. I'll assume you know what layers, attention and the feed-forward network are. If those are new to you, start with the basics first. --- ## The problem: every weight, every token Start with a normal transformer. Researchers call it **dense**: every parameter takes part in processing every token. Ask it what 2 + 2 is, and all its weights do their bit, including the ones that hold chemistry, French grammar and Python. Each transformer layer has two main parts. **Attention** gathers context from other tokens. The **feed-forward network** (FFN), a small two-layer network applied to each token on its own, is where researchers think much of the model's stored knowledge lives. And the FFN is big. In the standard design, the FFN in each layer has about twice as many weights as the attention part, so roughly two-thirds of the layer's weights sit in the FFN. Here's the bind. To make a model know more, you add parameters. In a dense model, every parameter you add must run on every token. So twice the parameters means roughly twice the compute for every word of every answer, in training and in use. **More knowledge means slower and more expensive.** What if most of that knowledge could stay idle until it's needed? --- ## How it works: a router and some experts The mixture-of-experts idea is simple to state: 1. **Cut the big FFN into several smaller FFNs.** Each one is called an **expert**. In Mixtral, every layer has 8 experts. 2. **Add a router.** The router (also called a gating network) is a tiny network that looks at each token's current vector and gives every expert a score. 3. **Pick the top k.** Only the experts with the highest scores run. Mixtral uses **top-2**: the two best experts process the token, and their outputs are blended, weighted by the router's scores. The other six do nothing for that token. Attention stays shared. Only the FFN part is split into experts. That's why this kind of model is called **sparse**: for any one token, most of the network is switched off.

A dense layer runs its whole feed-forward block for every token; an MoE layer runs a small router, then only the top two of eight experts.

### An analogy, and where it breaks Think of a hospital with a triage desk. You don't get seen by every doctor in the building. A nurse looks at you for a moment and sends you to the two who seem to fit best. The hospital employs dozens of specialists, but each patient only costs a couple of appointments. Where it breaks: in a hospital, the triage nurse knows that the cardiologist does hearts. In an MoE model, nobody assigns specialties. The router and the experts are trained together from scratch, and whatever division of labour emerges is whatever helped the model predict the next token. As we'll see, it doesn't look much like medical departments. ### It happens in every layer Mixtral has 32 layers, and every one has its own router and its own 8 experts. So one token might go to experts 3 and 7 in layer 1, experts 1 and 5 in layer 2, and so on. The next token takes a different path through the stack. That's where the two numbers come from: - **Total parameters (46.7B):** everything stored, all experts in all layers plus the shared parts. - **Active parameters (12.9B):** what one token actually flows through, two experts per layer plus attention and the other shared parts. Why isn't it 8 × 7B = 56B, as the name suggests? Because the "7B" names the size of a whole Mistral 7B-style model, and only the FFN parts are copied eight times. Attention and embeddings are shared. The Mixtral paper puts it simply: each token "has access to 47B parameters, but only uses 13B active parameters". --- ## A short history: 1991 to a trillion parameters The idea is older than most people in AI today. **1991.** Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in *Neural Computation*. They trained several small networks plus a gating network that learned which expert should handle which input. On a vowel-recognition task, the system split the problem into sub-tasks, each handled by a very simple expert. In that early version, every expert ran on every input and the gate decided how much to trust each one. Running only a few experts per input came later. **2017.** A Google team led by Noam Shazeer (with Geoffrey Hinton and Jeff Dean among the authors) published "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer". They put a sparse MoE layer between layers of an LSTM language model (transformers had not been published yet), with a **noisy top-k** router that picked just a few experts per input. Their biggest layer had 131,072 experts and up to 137 billion parameters. They reported "greater than 1000x improvements in model capacity with only minor losses in computational efficiency". **2021.** Google's **Switch Transformer** (Fedus, Zoph and Shazeer) brought MoE into transformers at scale and made routing even simpler: **top-1**, each token goes to a single expert. Built on Google's T5 models, it reached up to 7x faster pre-training with the same compute. Their largest model, Switch-C, had about 1.6 trillion parameters and 2,048 experts. **2023 onward.** MoE moved from research into widely used models. Mixtral made it famous among developers. In February 2024 Google said its Gemini 1.5 used a mixture-of-experts architecture. DeepSeek-V3 (December 2024) has **671 billion total parameters, with 37 billion active per token**, using 256 small routed experts plus one shared expert per MoE layer, with 8 routed experts chosen per token. Meta's Llama 4 Maverick (April 2025) has **400 billion total and 17 billion active**, with 128 routed experts plus a shared expert; each token goes to the shared expert and one routed expert. Notice the trend: **more, smaller experts, plus a shared expert** that every token uses. The shared expert can hold common knowledge every token needs, so the routed experts don't each have to relearn it. What about closed models like GPT-4? There have been many rumours. The makers of most closed models haven't published their architectures, so I won't claim anything about them here. --- ## The traffic jam: load balancing There's a catch, and it took years of engineering to manage. The router learns, like the rest of the model. Left alone, it develops favourites. The 2017 paper described it precisely: the gating network "tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network." Picture a supermarket with eight checkouts where everyone queues at the same two. Six cashiers stand idle, and the line is long. In a model, idle experts are wasted parameters, and the overworked ones become a bottleneck. On real hardware it's worse. In large MoE models, experts are often spread across different chips, a setup called **expert parallelism**. Each expert gets a fixed **capacity**, a maximum number of tokens it can take per batch. In the Switch Transformer, when too many tokens chose the same expert, the overflow tokens were **dropped**: in the paper's words, "computation is skipped and the token representation is passed directly to the next layer through the residual connection". The token didn't crash anything, but that layer's expert never processed it. The fixes: - **An auxiliary loss.** Add a small extra penalty during training when routing is uneven. It nudges the router to spread tokens more evenly. Shazeer's 2017 paper used a version of this, and the Switch Transformer used a simpler one. The downside: it's a second goal pulling against the main one, and if it's too strong it can hurt quality. - **A capacity buffer.** Give each expert a bit of slack (the "capacity factor"), so fewer tokens overflow, at the cost of more memory and compute. - **Balancing without the extra loss.** DeepSeek-V3 introduced what it called an "auxiliary-loss-free strategy for load balancing". Each expert gets a bias number that's added to its router score only for choosing experts. If an expert is overloaded, its bias is nudged down; if underloaded, nudged up. DeepSeek reported that, with this and its other measures, V3 drops no tokens during training or inference. Keeping the queues even is a large part of what makes MoE work in practice. --- ## What are the experts experts in? Now the question everyone asks. If there are eight experts, surely one handles maths, one code, one French, one poetry? That's what the name suggests. It's mostly not what happens. The Mixtral paper tested it. The team ran text from very different sources through the model and looked at which experts were chosen: maths and physics papers from arXiv, medical abstracts from PubMed, and philosophy papers from PhilPapers. The expert choices looked very similar across all of them. In their words: **"we do not observe obvious patterns in the assignment of experts based on the topic."** The one slight exception they noted was a dataset of maths problems (DM Mathematics), whose expert distribution was "marginally different", which they put down to that dataset being synthetic and narrow. What they did see was more about the **shape** of text than its meaning: - Words like "self" in Python and "Question" in English often went through the same expert. - In code, the indentation tokens were "always assigned to the same experts". - Consecutive tokens often went to the same experts in middle and later layers. At layer 15, the same expert showed up as first or second choice for consecutive tokens roughly 62 to 67% of the time, far above chance. So the experts aren't specialists in the human sense. They're more like workers who've each become good at certain kinds of tokens and contexts, in a split nobody designed and that's hard to name. One caution: this is one analysis of one model. Models with many more, smaller experts, like DeepSeek-V3, may divide the work differently, and researchers are still studying it. (If you want to know how people look for meaning inside models, that's the subject of [part 1 on interpretability](https://phamphutai.com/2026/10/20/interpretability-looking-inside-an-ai/).) The honest summary: "mixture of experts" is a slightly misleading name. **It's less a panel of specialists and more one big model that has learned to switch most of itself off for each token.** --- ## The memory catch: total vs active So why isn't every model built this way? Because idle experts still take up space. The router can choose any expert for any token at any moment. So **every expert must be loaded and ready in memory**, even though only a few run at a time. The Mixtral paper says it directly: memory costs for serving Mixtral are "proportional to its sparse parameter count, 47B". Meta says the same about Llama 4: "all parameters are stored in memory", while only a subset is used for each token.

MoE models run like small models but must be stored like big ones: the outlined bar is what has to fit in memory.

That's the trade: - **Compute per token** follows the active count. Mixtral runs at roughly the cost of a 13B model. - **Memory** follows the total count. You still need room for all 47B. For a company serving millions of users on racks of GPUs, that's a great deal. The memory is paid once, and every token is cheaper to produce. For someone trying to run a model on a laptop or a single consumer GPU, it's a harder one: the thing that has to fit is the big number. There's also extra engineering: spreading experts across chips means tokens must be shipped between them, and that communication can become the bottleneck. A smaller catch: because the experts see fewer tokens each during training, MoE models have historically been trickier to train and fine-tune stably. The Switch Transformer paper names training instability as one of the main problems it had to address. Modern recipes have improved, but it's one reason MoE isn't free. --- ## What it means in practice If you build with AI or choose models, here's what I'd keep in mind. **1. Ask "total or active?"** When you see a parameter count, check which one it is. A 400B MoE model with 17B active is a very different beast from a 400B dense model. It may answer at something like a small model's speed and cost per token, but need a big model's hardware. **2. For self-hosting, the total is what has to fit.** If you plan to run an open model on your own hardware, size the GPU memory for the total parameters (plus the working memory for long prompts, which is the topic of part 3), not the active count. Shrinking the weights with quantization helps, and that's part 4. **3. Compare models on your tasks, not on architecture.** MoE is an efficiency trick, not a quality guarantee. A dense model and an MoE model of similar cost can both be the right choice. The only way to know is to test them on your own work. Those [hidden costs nobody shows in the demo](https://phamphutai.com/2026/10/14/hidden-costs-of-ai-tools/) include the hardware bill, and MoE changes its shape. **4. Don't take "expert" literally.** There's no maths department inside. If a product claims its model "sends your legal question to the legal expert", that's marketing language, not how MoE routing works, at least in the models whose routing has been studied publicly. --- ## Key ideas - A **dense** model runs every parameter for every token, so more knowledge means more compute per answer. - A **mixture-of-experts** model splits each layer's feed-forward block into **experts**, and a small **router** picks the **top k** (Mixtral: 2 of 8) for each token, in every layer. - **Total parameters** are what's stored; **active parameters** are what runs per token. Mixtral: 46.7B total, 12.9B active. DeepSeek-V3: 671B and 37B. - The idea dates to **1991**, was scaled by Google in **2017** (sparse top-k gating) and **2021** (Switch Transformer, top-1, about 1.6 trillion parameters). - Routers tend to play favourites, so MoE needs **load balancing**: auxiliary losses, capacity limits, or DeepSeek's loss-free bias trick. - In Mixtral, experts split by **token patterns and syntax, not topics**. - **All experts must sit in memory**: MoE is cheap per token but needs big hardware. *Sources: Simon Willison, "Many options for running Mistral models in your terminal using LLM" (18 Dec 2023), [simonwillison.net](https://simonwillison.net/2023/Dec/18/mistral/); Mistral AI, "Mixtral of experts" (11 Dec 2023), [mistral.ai/news/mixtral-of-experts](https://mistral.ai/news/mixtral-of-experts); Jiang et al., "Mixtral of Experts" (Jan 2024), [arXiv 2401.04088](https://arxiv.org/abs/2401.04088); Jacobs, Jordan, Nowlan and Hinton, "Adaptive Mixtures of Local Experts", Neural Computation 3(1), 1991, [doi 10.1162/neco.1991.3.1.79](https://doi.org/10.1162/neco.1991.3.1.79); Shazeer et al., "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (2017), [arXiv 1701.06538](https://arxiv.org/abs/1701.06538); Fedus, Zoph and Shazeer, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity" (2021), [arXiv 2101.03961](https://arxiv.org/abs/2101.03961); DeepSeek-AI, "DeepSeek-V3 Technical Report" (Dec 2024), [arXiv 2412.19437](https://arxiv.org/abs/2412.19437); Meta AI, "The Llama 4 herd" (5 Apr 2025), [ai.meta.com](https://ai.meta.com/blog/llama-4-multimodal-intelligence/); Google, "Our next-generation model: Gemini 1.5" (15 Feb 2024), [blog.google](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/); Vaswani et al., "Attention Is All You Need" (2017), [arXiv 1706.03762](https://arxiv.org/abs/1706.03762), for the standard feed-forward size.* *Related: previous in the series, [Interpretability: How Scientists Look Inside an AI's Mind](https://phamphutai.com/2026/10/20/interpretability-looking-inside-an-ai/), next, [Long Context and the KV Cache: How AI Reads a Whole Book at Once](https://phamphutai.com/2026/10/20/long-context-and-the-kv-cache/), and [The Hidden Costs of AI Tools Nobody Shows in the Demo](https://phamphutai.com/2026/10/14/hidden-costs-of-ai-tools/).*