What is speculative decoding and when does it make LLM inference faster?

Speculative decoding speeds up a large language model (LLM) by drafting the next few tokens cheaply and letting the model check all of them in one step. The same model answers faster on the same hardware, which reduces how long users wait for a response.

Providers can make a model faster in two different ways. Quantization cuts the precision of model weights, and distillation trains a smaller model to imitate a larger one. Both methods make the model cheaper to run and can change how the model answers your prompts.

Speculative decoding leaves the model’s weights untouched and changes only how tokens get generated. You should know which kind of speedup a provider uses before you compare speed claims.

How does speculative decoding work?

An LLM normally produces one token per forward pass through the full model. Each pass reads every model weight from GPU memory, so a 500-token answer takes 500 full reads. Many of those tokens are easy to predict from the surrounding text: a closing bracket in code, the word “States” after “United,” or repeating a variable already defined in the prompt.

Speculative decoding uses those easy tokens to skip full passes through the large model. In its standard form, speculative decoding runs two models. A small draft model guesses the next few tokens and gets the easy ones right most of the time. The large target model you called then checks all of those guesses in one forward pass, which costs about as much as producing a single token. Every guess the target model accepts is a token it didn’t need a separate pass to produce.

Here’s how it works:

1. The draft model generates k candidate tokens one at a time, using k forward passes. The serving engine sets k, and small values such as 3 to 5 are typical.
2. The target model scores all k candidates in parallel in one forward pass.
3. A verification rule checks the candidates from left to right against the target model’s predictions.
4. The engine keeps every candidate before the first rejection and discards the rejected candidate along with every candidate after it.
5. The target model adds one token of its own from the same pass. If a candidate was rejected, the target model’s token replaces it. If every candidate was accepted, the target model’s token extends the response by one more.

Suppose the draft model proposes four tokens to finish a line of code, return, total, ; and }. If the target model accepts all four guesses, one pass produces five tokens (the four drafts plus the extra token). Normal decoding would require five sequential target-model passes. The actual speedup depends on drafting and verification overhead.

Now suppose the target model rejects the first guess. The engine discards all four guesses, and the pass produces only the target model’s own token for that position. One pass yields one token, which matches what normal decoding gets from one pass. That round runs slightly slower than normal decoding, because the draft model spent time writing four guesses the engine threw away.

The engine repeats these rounds until the response is complete. Many production systems replace the separate draft model with EAGLE-3, a small draft head that reads the target model’s internal features. N-gram decoding and multi-token prediction (MTP) follow the same draft-and-verify loop without a separate draft model.

What is n-gram speculative decoding?

N-gram speculative decoding looks for matching token sequences in your prompt and proposes the text following them as guesses, without running a second model. Some engines call this method prompt lookup decoding. It works best when the output repeats the input text, such as summaries that reuse source wording or code edits that preserve existing code.

Is MTP speculative decoding?

Yes, MTP is considered speculative decoding. DeepSeek-V3 and other models trained with multi-token prediction include extra prediction modules that can draft the next few tokens while the main model then verifies those drafts. The drafter ships inside the model, so a provider can run speculative decoding without loading a separate draft model. vLLM’s MTP documentation explains how to enable it in vLLM.

What is the acceptance rate in speculative decoding?

The acceptance rate is the share of drafted tokens the target model accepts during verification. It decides whether speculative decoding makes a response faster or slower. When the acceptance rate is high, the target model keeps most of the draft model’s guesses, so each target model pass produces several tokens instead of one.

When the acceptance rate is low, the target model rejects most guesses early in each round, so each pass produces about one token, the same as normal decoding. The draft model still spends time writing the rejected guesses. At a low enough acceptance rate, that wasted drafting time makes responses arrive slower than they would without speculative decoding.

When does speculative decoding speed things up?

Speculative decoding speeds things up when the acceptance rate is high enough to save more time than drafting and checking cost. Five conditions decide whether that happens:

1. Drafter needs to be both fast and accurate. A tiny drafter finishes its guesses quickly but gets more of them rejected. A larger drafter gets more guesses accepted but takes longer to produce them. The best drafter produces the most accepted tokens per second.
2. Speculation helps most when the GPU is waiting on memory. With few requests in flight, each decoding step spends most of its time reading the model weights from GPU memory while the compute units sit mostly idle. Verifying several draft tokens in one pass spreads the cost of reading model weights across multiple accepted tokens. With larger batches, more of the GPU’s compute capacity is already in use, so the extra verification work may offer less speedup.
3. KV cache can be the main bottleneck during decoding with long contexts. The KV cache stores attention data for every earlier token in the context. Once a context reaches tens of thousands of tokens, reading the KV cache can take longer than reading the model weights, even in a large batch. Checking several drafted tokens reuses each KV cache read, so speculation keeps paying off.
4. The workload changes how many guesses get accepted. Code and math contain more predictable tokens than open-ended conversation, so drafters get more guesses accepted on those tasks. A speedup measured on one task does not carry over to another.
5. Serving configuration affects speedup gain. Running the drafter, building tree drafts that test several possible continuations at once, checking them and scheduling requests all add overhead to each step. The engine’s configuration decides whether that overhead eats the speedup. Compare time per output token and total output throughput with speculation on and off at your expected concurrency.

The experiments below illustrate these conditions. Each result is relative to the study’s baseline without speculative decoding; the models, hardware, workloads, and measurement methods differ, so they are not directly comparable.

Does speculative decoding change the output?

Standard speculative decoding is lossless. Which means it doesn’t change what the target model is likely to say. Under greedy decoding, every accepted token is the token the target model would have picked on its own. Under random sampling, the verification rule accepts or replaces each guess so that every token appears with the same probability the target model alone would give it. The target model’s weights stay unchanged.

Lossless does not promise identical text on every run. With random sampling, two runs can produce different wording. Even with greedy decoding, a change in batch size changes the order in which the GPU adds numbers. The resulting rounding differences can flip a near-tie between two tokens, as explained in vLLM’s documentation.

However these differences are small in practice. In a DeepMind experiment, Chinchilla 70B with a 4B draft model on 16 TPU v4 chips summarized XSum articles about twice as fast at batch size 1 with greedy decoding. Its ROUGE-2 summary-quality score moved only from 0.157 to 0.156.

A provider can advertise speculative decoding and still change your answers in two ways. Weight quantization stores model weights at lower precision, which can reduce accuracy. Providers can also relax speculative verification to pursue more speed, giving up the lossless guarantee. SGLang exposes acceptance thresholds that allow more permissive checks, while vLLM explicitly distinguishes its standard verification from lossy alternatives.

What to ask before choosing a provider

Two questions tell you whether a provider’s speedup comes from strict speculative decoding on the model you chose.

• Is the model you call quantized, or does it run at its original precision?
• Does the provider’s speculative decoding use strict verification?

If the provider runs the original model with strict verification, the speedup is real and the model’s answers stay the same. How large that speedup is then depends on what the model is generating and how many requests it serves at once. For a measured example, Making an EAGLE fly: How We Got 2.6x Faster LLM Inference (Without Cheating) shows how EAGLE-3 speedups change across tasks and concurrency levels.