A plain ("dense") transformer produces each new token with one full forward pass: every layer, every weight, in order. Then it appends that token and does it all again. Tokens come out strictly one after another.
The surprising part is that the bottleneck is usually not the math. Generating for one user, the GPU spends most of its time reading the weights out of memory:
So there are three ways to spend less per token, and each technique below takes one of them:
Recall that the MLP is where most parameters, and most stored knowledge, live. MoE replaces the single MLP in each layer with E separate MLPs ("experts"), plus a tiny router: one small matrix that scores every expert for the current token. Only the top k experts run, and their outputs are blended by those scores. Attention stays dense.
Click a row to see that token's blend. Router weights are hand-set to be readable (illustrative). In trained models, experts specialize less cleanly than this, and more by syntax and token type than by topic.
Load-balancing term added to the training loss (Switch Transformer), where fi is the fraction of tokens sent to expert i and Pi is its mean router probability:
It is smallest when the load is even. Without it, the router "collapses": a few experts get all the tokens, train more, get picked more, and the rest die. Toggle router → no balancing above to see this. DeepSeek-V3 instead nudges a per-expert bias up or down to even out load.
| Model | Experts (active) | Total | Active |
|---|---|---|---|
| Mixtral 8×7B | 8 (2) | 47B | 13B |
| DeepSeek-V3 | 256 + 1 shared (8) | 671B | 37B |
| Kimi K2 | 384 + 1 shared (8) | ~1T | 32B |
Knowledge capacity scales with total parameters. Cost per token scales with active parameters.
The residual stream holds the model's current best guess, and it is refined layer by layer. You can read that guess at any layer by applying the final norm and the output head early. This is called the logit lens. After "The capital of", the token "of" is obvious by layer 2. "Seine" needs the deep layers that recall facts. Early exit stops a token as soon as its guess is confident enough.
Each row predicts the next word of "The capital of France is Paris, and its river is the Seine." The layer-by-layer confidences are illustrative, shaped like real logit-lens measurements, not taken from a trained model. Drag the threshold down: compute drops, then wrong early guesses start to slip through, starting with "of → the" and "and → the".
hℓ is the residual stream after layer ℓ, and WU is the output head. Emit argmax pℓ* and skip layers ℓ*+1…L. Real systems (e.g. CALM, 2022) train the model so intermediate layers give good predictions, and calibrate τ so outputs stay close to the full model's.
| Skips | Saves | Costs | Same output? | |
|---|---|---|---|---|
| Mixture of experts | most MLP weights, per token (width) | compute per token; more knowledge per FLOP | memory for all experts, routing, cross-GPU traffic | no: it is a different model, trained that way |
| Early exit | later layers, for easy tokens (depth) | compute and latency on easy tokens | KV-cache holes, poor batching, some errors | approximately |
| Speculative decoding | sequential big-model passes (time) | latency, often 2–3× | draft model, wasted guesses | yes, exactly the same distribution |
They stack: a production system can serve an MoE model with speculative decoding and quantized weights all at once.
Remember the cost is reading the weights. One big-model pass over 5 tokens reads the weights once, so it costs about the same as a pass over 1 token. So a small draft model guesses the next γ tokens one by one (cheap). Then the big target model scores all γ guesses in a single parallel pass, exactly like training-time teacher forcing. It keeps the longest acceptable prefix, and a clever accept/reject rule makes the output distribution exactly the big model's. Nothing is approximated.
Both models are toy word-level models built from a small set of sentences. The target is a bigram model; the draft is the target blurred toward uniform by the "draft quality" knob. The accept/reject algorithm is the real one (Leviathan et al. and Chen et al., 2023).
For each drafted token x ~ q (draft), accept with probability
At the first rejection, resample from the leftover mass where the target wanted more than the draft gave, and stop:
Why it equals p: tokens that are accepted contribute min(p,q). Rejections happen with probability 1−β (β = Σ min(p,q)) and the resample refills exactly (p−q)+:
Sampling is exact even when the draft is bad: a bad draft only costs speed. Set draft quality to 0 and run again.