Mixture of Experts
Hold many parameters, use a few per token. How models grew far larger without getting proportionally slower.
On this page
In a standard transformer, every token passes through every parameter. Doubling the parameters roughly doubles the compute per token.
Mixture of experts breaks that link. The model holds many parameters and routes each token through only a small fraction of them.
The mechanism
Replace one feedforward layer with several parallel copies — the experts — plus a small router network.
For each token, the router scores every expert and selects the top few, commonly two. Only those run. Their outputs are combined, weighted by the router’s scores.
So a layer with 8 experts holds 8× the parameters of a dense layer, while each token activates 2 of them. Capacity grows 8×; compute per token grows about 2×.
This is why total and active parameter counts are different numbers. A model advertising 400B total might activate 40B per token, and its speed and memory profiles come from different halves of that: memory from the total (all experts must be resident), speed from the active count.
Why it works
Parameters store patterns. Not every pattern is relevant to every token. Code tokens do not need the parameters that handle Portuguese poetry.
A dense model forces every token through everything, which is wasteful. Routing lets the model specialize — different experts handle different kinds of input — and each token pays only for what it uses.
Specialization is not assigned. It emerges during training because it reduces loss. Inspection shows experts developing preferences for particular languages, syntactic patterns, or domains, though the divisions are messier than the word “expert” suggests.
The difficulties
Load imbalance is the central training problem. Nothing inherently prevents the router from sending most tokens to a few favourite experts, leaving others undertrained and wasting capacity. Training therefore includes an auxiliary loss that penalizes uneven routing, encouraging balanced use. Getting this balance right without harming quality is much of the engineering.
Routing is discrete and therefore awkward to train. Selecting the top-k experts is not differentiable, so the gradient reaches the router only through the weighting of chosen experts. Various approximations exist; none are clean.
Memory does not shrink. All experts must be in memory even though most are idle for any given token. An MoE model needs the memory of its total parameter count while delivering the speed of its active count — which is a good trade when memory is available and a bad one when it is not.
Distributed serving is harder. Experts are often spread across devices, so routing means communication between them. Batch composition affects which experts are hit, making latency less predictable than a dense model’s.
Fine-tuning is trickier. Routing behaviour can shift during fine-tuning, degrading the specialization that made the model good.
What it means practically
Read both parameter numbers. “400B total, 40B active” describes cost far better than either figure alone. Comparing an MoE model’s total count against a dense model’s is comparing different things.
MoE models are strong per unit of inference compute — that is the point. They are less impressive per unit of memory.
Self-hosting is harder than the active count suggests. The memory requirement is set by total parameters, so a model that generates as fast as a 40B dense model may need the hardware of a 400B one.
Latency variance is higher. For strict latency requirements, this is worth measuring rather than assuming.
Relation to other efficiency work
MoE saves compute per token. Quantization saves memory per parameter. Speculative decoding saves sequential steps. The three are independent and combine — production systems often use all of them.
Worth distinguishing sparsity of this kind, where the architecture routes conditionally, from sparsity in attention patterns, where the sequence is attended to selectively. Different mechanisms, different savings.
What to remember
- MoE replaces a feedforward layer with many experts plus a router that activates only a few per token.
- Capacity scales with total parameters while compute scales with active parameters — decoupling size from speed.
- Specialization emerges from training; load balancing requires an explicit auxiliary loss.
- Memory is set by total parameters, so self-hosting is harder than the active count implies.
- Always read both parameter numbers; they answer different questions.