Speculative Decoding

The idea in one minute#

Decoding is slow because it is sequential: one forward pass, one token. But a forward pass can check several tokens as cheaply as it can produce one, because checking is parallel. Speculative decoding exploits that. Something cheap proposes the next few tokens — a small model, an extra head on the big model, or simply a guess that the text will repeat something already in the prompt. The big model then runs once over all the proposals and verifies them. Every proposal it agrees with is a token gained for free. The output is exactly what the big model would have produced alone. In vLLM this is not a bolt-on: draft tokens are ordinary scheduled tokens with a rollback, which is why the scheduler’s counters were designed the way they were.

A picture#

flowchart LR
  CTX["context: ... the cat sat"] --> P[":i-zap: <b>Proposer</b><br/><small>cheap: n-gram, EAGLE head,<br/>draft model, MTP</small>"]
  P -->|"drafts: on the mat ."| V[":nvidia: <b>Target model</b><br/><small>ONE forward pass over<br/>last token + 4 drafts</small>"]
  V --> R{"compare at each position"}
  R -->|"on ✓  the ✓  mat ✗"| ACC["accept 2 drafts<br/>+ the model's own token at the miss<br/><small>3 tokens from 1 pass</small>"]
  ACC --> RB["roll back the rejected positions<br/><small>num_computed_tokens −= 2</small>"]
  RB --> CTX
  class CTX neutral
  class P queue
  class V compute
  class R queue
  class ACC neutral
  class RB warn

How it really works#

Why verification is nearly free#

Decode is limited by memory bandwidth: each step reads all the model’s weights to produce one token per request, and the GPU’s arithmetic units are mostly idle (Prefill vs Decode). Feeding five tokens per request instead of one reads the weights the same number of times and uses some of that idle arithmetic. A step that verifies k drafts costs only a little more than a step that produces one token — until the GPU’s compute is actually saturated, which is exactly the caveat that follows.

The acceptance rule#

For greedy decoding the rule is simple: a draft token is accepted if it equals the token the target model ranks first at that position. The first mismatch ends acceptance, and the target model’s own choice at that position is kept. So every step yields at least one token and at most k + 1.

For random sampling, rejection sampling accepts a draft with a probability that depends on how much the target model agrees, and resamples on rejection in a way that leaves the output distribution identical to sampling from the target model alone. The scheduling is the same; only the comparison differs.

Either way: speculative decoding does not change what is generated. If quality drops with it enabled, that is a bug, not a trade-off.

Proposers#

Configured with --speculative-config, a JSON object:

vllm serve <target-model> \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 4,
                         "prompt_lookup_min": 2, "prompt_lookup_max": 5}'

The project’s own summary of when each pays off: model-based methods “provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.”

How the scheduler sees it#

Recall the comment at the top of schedule(): a request has num_computed_tokens and num_tokens_with_spec = prompt + output + spec tokens, and scheduling closes the gap. Drafts simply widen the gap.

Proposal. After a step, the proposer’s drafts are stored on the request as spec_token_ids. With async scheduling the scheduler does not have them yet, so it schedules placeholders (-1) and the worker, which ran the proposer, fills them in.

Scheduling. The gap is now 1 + k. The scheduler grants those tokens and records which ones are drafts in scheduled_spec_decode_tokens. It also asks the cache for extra room: num_lookahead_tokens reserves slots that a proposer with its own KV (such as EAGLE) will write.

Budget. Drafts consume the token budget like any other tokens. max_num_scheduled_tokens is set below max_num_batched_tokens so that drafts appended to the batch still fit.

Verification and rollback. In update_from_output:

num_draft_tokens = len(scheduled_spec_token_ids)
num_sampled = self.num_sampled_tokens_per_step
num_accepted = max(len(generated_token_ids) - num_sampled, 0)
num_rejected = num_draft_tokens - num_accepted
# Rejections roll back num_computed_tokens ...
if request.num_computed_tokens > 0:
    request.num_computed_tokens -= num_rejected

The scheduler had optimistically counted all 1 + k positions as computed. Rejected positions are subtracted again. Their KV entries are garbage, but nothing needs to be erased: the slots will simply be overwritten when the correct tokens are computed.

Caching. Blocks receive hashes only up to request.num_tokens, never including drafts (Allocating Slots), so a rejected draft can never leak into the prefix cache.

The batch shape stays uniform#

Full CUDA graphs need a uniform batch: every request contributing the same number of tokens (CUDA Graphs and Compilation). With speculation that number is 1 + k. A request that has just finished its prompt has no drafts yet and would contribute only one token, breaking uniformity. So the scheduler pads it:

# Pad new decode requests to uniform spec decoding size to
# preserve full cudagraph for this step.
...
padded_num_tokens = 1 + self.num_spec_tokens
...
scheduled_spec_decode_tokens[request_id] = [-1] * self.num_spec_tokens

It even prefers not to schedule the request this step over scheduling it unpadded: “Prefer to not schedule than schedule un-padded.” A few wasted rows are cheaper than losing the recorded graph for the whole batch.

When it helps and when it hurts#

Let α be the probability that each successive draft is accepted. The expected number of tokens per step with k drafts is

E[tokens] = 1 + α + α² + … + α^k = (1 − α^(k+1)) / (1 − α)

and the step costs more than a plain one: the proposer must run, and the target processes k + 1 rows per request.

The documentation is explicit that this is a tool to “reduce inter-token latency under medium-to-low QPS … memory-bound workloads.” Two mechanisms on main address the high-load case:

• Dynamic speculative decoding chooses k from the current batch size each step (num_spec_tokens_to_schedule in the SchedulerOutput): many drafts when the server is quiet, few or none when it is busy.
• Adaptive verification sizes each request’s verification from the drafter’s confidence.

Measuring it#

Acceptance rate is accepted ÷ proposed; mean acceptance length is 1 + accepted ÷ drafts and is the number that maps directly to speedup. If acceptance at position 4 is 10%, a k of 3 would cost less and gain almost as much. Per-request acceptance figures can also be returned in responses.

Interactions#

• Structured output. Drafts must satisfy the grammar too; forbidden drafts are invalidated before verification and excluded from the acceptance statistics.
• Async scheduling. Supported for the model-based methods and GPU n-gram; plain ngram and suffix turn async scheduling off, because their drafts are produced on the CPU in the engine core and cannot be filled in by the worker.
• Stop conditions. If an accepted run of tokens contains end-of-sequence, tokens after it are trimmed.
• Tensor parallelism. A draft model can use a different tensor-parallel size from its target (draft_tensor_parallel_size).

Code#

Expected speedup as a function of acceptance rate, draft count and load. The cost model has two parts: running the proposer, and the extra rows the target must process, which are almost free when the GPU has spare compute and expensive when it does not.

package main

import (
	"fmt"
	"math"
)

// expectedTokens per step with k drafts, each accepted with probability alpha
// given that all earlier ones were accepted.
func expectedTokens(alpha float64, k int) float64 {
	if alpha == 1 {
		return float64(k + 1)
	}
	return (1 - math.Pow(alpha, float64(k+1))) / (1 - alpha)
}

// stepCost relative to a plain decode step (= 1.0).
// draft:   cost of proposing one token, as a fraction of a target step.
// rowCost: marginal cost of one extra row in the target's batch.
//
//	~0.03 when compute is idle, ~1.0 when the GPU is compute-saturated.
func stepCost(k int, draft, rowCost float64) float64 {
	return 1 + float64(k)*draft + float64(k)*rowCost
}

func main() {
	type regime struct {
		name    string
		rowCost float64
	}
	regimes := []regime{{"low load (compute idle)", 0.03}, {"high load (compute saturated)", 0.60}}
	const draft = 0.05 // an EAGLE-style head: a few percent of a target step per token

	for _, rg := range regimes {
		fmt.Printf("%s\n", rg.name)
		fmt.Printf("  %-12s", "acceptance")
		for _, k := range []int{1, 2, 3, 5, 8} {
			fmt.Printf("   k=%d  ", k)
		}
		fmt.Println("  best k")
		for _, alpha := range []float64{0.4, 0.6, 0.8, 0.9} {
			fmt.Printf("  %-12.1f", alpha)
			bestK, best := 0, 1.0
			for _, k := range []int{1, 2, 3, 5, 8} {
				s := expectedTokens(alpha, k) / stepCost(k, draft, rg.rowCost)
				fmt.Printf("  %5.2fx", s)
				if s > best {
					best, bestK = s, k
				}
			}
			if bestK == 0 {
				fmt.Println("  none: slower than baseline")
			} else {
				fmt.Printf("  %d\n", bestK)
			}
		}
		fmt.Println()
	}
}

Under low load every row is a speedup and more drafts are better when acceptance is high. Under high load most cells are below 1.0: the same configuration that halved latency at midnight reduces throughput at noon. That table is the argument for choosing k dynamically.

Remember this#

• A proposer guesses k tokens; the target verifies them in one pass; output is unchanged.
• Each step yields between 1 and k + 1 tokens. Expected yield is (1 − α^(k+1)) / (1 − α).
• In the scheduler, drafts are extra scheduled tokens; rejected ones are rolled back by decrementing num_computed_tokens.
• Drafts are never given block hashes, so they cannot enter the prefix cache.
• Batches are padded to 1 + k tokens per request to keep full CUDA graphs.
• It lowers latency when the GPU has spare compute and can lower throughput when it does not.
• Watch acceptance per position; size k to where acceptance is still high.

Try it#

1. In the program, model a full draft model by setting draft to 0.25. How does the best k change at acceptance 0.8 under low load?
2. Find the acceptance rate at which k = 3 breaks even under high load.
3. On a real server, enable ngram speculation and send (a) a request to rewrite a long code file with one small change, and (b) a request for a poem. Compare spec_decode_num_accepted_tokens for the two.

Check yourself#

1. Why does verifying several draft tokens cost much less than generating them one at a time?
2. What does the scheduler do to a request’s counters when two of four draft tokens are rejected?
3. Why can speculative decoding reduce throughput on a heavily loaded server?

Sources#

Checked on 5 October 2026 against vLLM v0.30.0 and main at commit 0c16eee.

• Speculative decoding — methods, the selection table, the config schema
• vllm/v1/core/sched/scheduler.py — padding, rollback, dynamic k
• vllm/v1/spec_decode/ — the proposers and metrics
• vllm/config/vllm.py — which methods keep async scheduling
• Fast Inference from Transformers via Speculative Decoding