Speculative Decoding
Guess several tokens with a small model, verify them all at once with the large one. Free speed, identical output.
On this page
Generation is sequential: one token at a time, each requiring a full pass through the model. That pass is dominated by reading every parameter from memory, not by arithmetic.
Which means the hardware is mostly idle. There is compute capacity available; the bottleneck is memory bandwidth for a single token’s worth of work.
Speculative decoding exploits that gap. Verifying several tokens costs almost the same as generating one, because verification is parallel and hits the same memory once.
How it works
Two models: a small fast draft model and the large target model you actually want output from.
- The draft model generates several tokens ahead — say four — cheaply and sequentially.
- The target model processes all four in a single forward pass, producing its own probability distribution at each position.
- Compare. Accept every drafted token that matches what the target would have produced; reject from the first mismatch onward.
- Continue from the last accepted position.
If all four are accepted, you got four tokens for roughly the cost of one target pass. If the first is rejected, you got one token — the target’s own — and wasted the draft work.
Output is identical
The property that makes this worth using rather than merely interesting: the output distribution is provably unchanged.
The acceptance rule is constructed so the resulting token sequence has exactly the distribution the target model would have produced alone. Rejected positions fall back to sampling from the target’s distribution, adjusted so no bias is introduced.
This is not an approximation and not a quality tradeoff. It is the same output, faster. That distinguishes it from quantization or distillation, which both trade quality for speed.
What determines the gain
Acceptance rate is everything. It depends on how well the draft model predicts the target’s choices, and that varies enormously by content — boilerplate, common phrasing, and predictable code are drafted accurately; novel reasoning is not.
Draft length trades off. Longer drafts mean more tokens per verification when accepted, and more wasted work when rejected early. Optimal length depends on acceptance rate, and adaptive schemes adjust it during generation.
Draft model cost must stay small relative to the target. A draft model too close in size eats the savings.
Typical reported speedups are roughly 2–3× on favourable content, less on hard generation. The variance is inherent — this technique speeds up predictable text most.
Variants
Independent draft model. A small model from the same family, chosen for vocabulary compatibility. Simple, and requires serving two models.
Self-speculation. Use a subset of the target’s own layers as the draft, skipping the rest. One model, no separate serving, generally lower acceptance.
Medusa-style extra heads. Add lightweight heads to the target that predict several positions ahead directly. No separate model, requires training the heads.
N-gram and lookup drafting. Draft by copying from the prompt or from a cache of recent text, with no model at all. Startlingly effective when output repeats input — summarization, editing, code modification. Cheap enough to be nearly free.
Where it does and does not help
Helps most on single-request latency: one user waiting for one response. This is the case it was designed for.
Helps least under heavy batching. When many requests are processed together, the hardware is already busy and the idle capacity speculation exploits is gone. Serving systems often disable it at high load — which is why a technique that halves latency for one user may do nothing for aggregate throughput.
Adds memory. A second model, or extra heads, plus draft KV cache.
Adds latency variance. Per-token timing becomes uneven depending on acceptance.
What to remember
- Generation is memory-bandwidth bound, leaving compute idle — verifying several tokens costs nearly the same as generating one.
- A small draft model proposes; the large target verifies in one pass; matching tokens are accepted.
- Output distribution is provably identical — speed with no quality cost, unlike quantization or distillation.
- Gains depend on acceptance rate, which is high on predictable text and low on novel reasoning.
- Best for single-request latency; largely ineffective under heavy batching.