Speculative Decoding

Speculative decoding lets a small draft model propose tokens that a larger target model verifies in one pass.

Standard autoregressive generation runs one target forward pass per token. A 100-token reply needs 100 sequential passes. Speculative decoding changes the loop. The draft model quickly writes a few candidates, typically 3-8. The target model then scores all of them in a single forward pass. Accepted tokens stay. The first rejected token forces regeneration from that point.

If the draft's distribution is close to the target's, most candidates pass and throughput jumps.

Verification is cheaper than generation: one target pass can check several tokens. Small models often guess what a large model would write on simple continuations. Speedups of 2-3x are common. Quality matches standard decoding because rejected tokens are regenerated correctly by the target. The draft can be a separate small model, a subset of the target's layers, or a simple n-gram model.

As models get larger, inference cost dominates, so this verification trick matters more.

Acceptance is not a string match. The target model’s probabilities decide whether a draft token is valid under the target distribution, often via a rejection sampling rule so the output law equals ordinary target decoding. If token 4 of 8 is rejected, tokens 1-3 stay and generation resumes from token 4.

A well-matched draft, including a distilled student or early-exit layers of the target, raises the accept rate. A weak n-gram draft still helps on boilerplate continuations. The 2-3x speedup assumes a decent accept rate; a bad draft wastes the verification pass.

Google's speculative decoding (2023) lets a small model draft tokens and a large model accept or reject them in a batch, which speeds inference.