Every model discussed in Parts 1–2 was implicitly dense: every parameter — every attention weight, every FFN weight — participates in every forward pass, for every token. That's the default, and it's simple to reason about. It's not the only option. Mixture-of-Experts (MoE) models replace the single FFN in each Transformer block with many FFNs ("experts") and a small router that picks a handful to run per token — everything else in that layer sits idle.
Dense: The Default
Recap from Part 1: each Transformer block has one feed-forward network (FFN), and every token's hidden vector passes through the entire thing:
FFN(x) = W2 · GeLU(W1 · x + b1) + b2 // one FFN, every token, every layer
Total parameters = active parameters. Every weight you pay to store, you also pay to compute against, for every token. Model size and inference compute cost are locked together — the only way to add capacity is to make the whole thing bigger, and every token pays for it.
Sparse: Not All Weights Have To Fire
"Sparse" is an overloaded word worth disambiguating — Part 2 already introduced one form of it:
Sparse attention — covered in Part 2. Each token only attends to a subset of other tokens (a window, a fixed pattern), not the whole sequence. Reduces the O(n²) cost of the attention calculation. All weights still exist and still run.
Sparse weights (pruning) — a trained dense model has some fraction of its individual weights zeroed out post-hoc (or during training) because they contribute little. Same architecture, fewer non-zero numbers to store or multiply.
Sparse activation (conditional computation) — entire subnetworks are skipped for a given token, decided dynamically per input rather than fixed in advance. This is what Mixture-of-Experts does, and it's the focus of the rest of this page.
Sparse attention shrinks the cost of looking at context. Sparse activation shrinks the cost of processing what you found. MoE is squarely the second kind.
Mixture-of-Experts: Many FFNs, Few Awake
An MoE layer swaps the single FFN for N separate expert FFNs (E1 … EN) plus a small router network. For each token, the router scores every expert, and only the top-k highest-scoring experts actually run:
G(x) = softmax(x · Wrouter) // score every expert
MoE(x) = Σi ∈ TopK(G(x)) G(x)i · Ei(x) // only run the top-k, weight by router score
Mixtral 8×7B, for example, has 8 experts per layer and routes each token to the top 2. Its total parameter count is roughly 47B, but only about 13B are actually multiplied against for any given token — the other 6 experts per layer simply don't run.
Total parameters
Everything stored on disk / in memory: every expert, every layer. This is your storage and VRAM bill — all experts must be loaded even though most sit idle on any given token.
Active parameters
What actually gets multiplied for a given token: router weights + the top-k experts it picked. This is your FLOPs bill — the thing that determines inference latency and training compute cost.
MoE's whole pitch: decouple "how much the model knows" (total parameters) from "how much compute it costs to answer" (active parameters). A dense model can't do that — for it, those two numbers are always the same.
The Routing Problem
The router is itself a small learned network, trained jointly with everything else — and left unconstrained, it tends to collapse: a few experts get routed most tokens and become very well-trained, while the rest are rarely selected and stay undertrained, making the router even less likely to pick them next time.
Load-balancing loss — an auxiliary training loss that explicitly penalises uneven routing, pushing the router toward spreading tokens more evenly across experts. Used by Switch Transformer, GShard, and most production MoEs.
Expert capacity limits — each expert is only allowed to accept a fixed number of tokens per batch; tokens that overflow a full expert get dropped (skip that layer) or rerouted. Bounds worst-case compute and memory per expert.
Noisy routing — adding noise to router logits during training encourages exploration, so promising but currently underused experts still get a chance to receive tokens and improve.
Why Bother? And What Does It Cost?
The appeal is real: for a fixed compute (FLOPs) budget, an MoE model can have dramatically more total parameters — and therefore capacity to memorise facts, cover more languages, specialise across domains — than a dense model of the same inference cost. That's why frontier-scale models increasingly lean sparse.
Dense — pros
Simple to train, simple to serve, predictable latency, no routing instability, no load-balancing tricks needed. Every FLOP you provision gets used on every token.
MoE — costs
All experts must be resident in memory even though few run per token, so VRAM requirements track total params, not active ones. Distributed serving needs expert parallelism (tokens get shuffled across devices to reach their chosen experts — real network overhead). Training is fussier: router collapse, load imbalance, harder reproducibility.
Dense trades flexibility for simplicity: one predictable cost, no routing to get wrong. MoE trades simplicity for a decoupled cost curve: more knowledge per FLOP spent answering, at the price of needing every expert in memory and a routing layer that has to be trained carefully, not just scaled up.
Note what doesn't change: attention still works exactly as in Part 1, and the lm_head/tokenisation choices from Part 2 are entirely independent of this decision. Dense vs. MoE is purely a statement about the feed-forward step — everything else in the Transformer skeleton stays the same.