Speculative Decoding in Production: Why Low QPS Speeds Up but High Load Slows Down

Background: Why Large Models Are Slow — It’s Not Just “The Model Is Too Big”

Large model inference is slow due to two main bottlenecks: prefill and decode.

Prefill encodes the entire input prompt into the KV Cache in one shot. The longer the input, the more noticeable the wait before the first token. Decode generates tokens one by one, reading a large amount of weights and KV Cache for each token. For interactive scenarios like online chat, code completion, and agent tool calls, users perceive not just total throughput but TTFT (Time To First Token) and ITL (Inter-Token Latency).

Speculative Decoding, also known as Assisted Generation, addresses the serial nature of the decode phase. Its core idea is straightforward: first, a faster “drafter” predicts the next few tokens, then the target large model verifies all these tokens at once. If multiple tokens are accepted, the target model effectively advances multiple generation steps with a single forward pass.

This is not simple caching or quantization. It’s more like changing “a large model writing an answer step by step” into “a small model writing a draft, and the large model reviewing it in batch.”

Core Principles: Draft, Verify, Acceptance Rate

A typical speculative decoding loop can be broken down into four steps:

1. Draft: The draft model or algorithm generates K candidate tokens. The drafter can be a small model, or structures like n-gram, suffix, Medusa head, EAGLE, or MTP.
2. Verify: The target model takes these K candidate tokens and verifies them in parallel in one or a few forward passes.
3. Accept / Reject: If the candidate tokens are acceptable under the target model’s current sampling rules, they are accepted; once an unacceptable token is encountered, the process falls back to the target model’s own output.
4. Repeat: The new context continues to the next round of drafting and verification.

The most critical variable here is not “how many tokens the draft guesses at once,” but the acceptance rate. If draft tokens are frequently accepted by the target model, the target model can produce multiple valid tokens in one forward pass, reducing ITL. Conversely, if the draft is often wrong, the system incurs the cost of drafting without reducing the number of target model calls, potentially increasing latency.

Hugging Face’s introduction to Assisted Generation explains this mechanism as: the assistant model first generates a candidate sequence token by token, and the target model then verifies all these candidates at once; the benefit comes from the target model confirming multiple tokens in a single forward pass, rather than generating one token at a time.

Key Engineering Judgment: It’s Not Faster for All Traffic

The vLLM documentation is cautious about speculative decoding’s positioning: it is suitable for reducing inter-token latency under low-to-medium QPS, memory-bound workloads. This qualification is crucial.

At low QPS, a single request’s decode often cannot fully utilize the GPU, and the serial cost of the target model generating one token per step is significant. Speculative decoding allows the target model to verify multiple candidate tokens at once, potentially improving the streaming output speed perceived by the user.

At high QPS, the situation changes. Continuous batching merges multiple requests, increasing GPU utilization, and a single request’s decode is no longer completely isolated. The paper “An Interpretable Latency Model for Speculative Decoding in LLM Serving,” submitted in May 2026, specifically analyzes this: the effective batch size in a production system varies with request rate, and the speedup from speculative decoding often decays as service load increases.

Therefore, speculative decoding is not a switch that makes things faster when turned on. It is more like a latency management tool that requires tuning based on traffic, model, scenario, and sampling strategy.

Comparison of Common Technical Approaches

1. N-gram / Prompt Lookup

N-gram speculation does not require an additional draft model. It finds repeated segments from the input prompt or generated content and uses potentially repeated tokens as drafts.

The TensorRT-LLM documentation points out that NGram directly copies draft tokens from the input prompt and previous output, making it suitable for scenarios with high n-gram overlap between input and output, such as summarization, document QA, multi-turn chat, and code editing.

Advantages: Simple deployment, low additional resource cost, low risk. Disadvantages: For open-ended creation, complex reasoning, and low-repetition tasks, the acceptance rate is often unstable.

2. Draft Model

The Draft Model is the most intuitive approach: use a small model to draft tokens for the large model.

The difficulty lies in the small model needing to be both fast enough and similar enough to the target model. Hugging Face’s Universal Assisted Generation article mentions that traditional assisted generation often requires the target and assistant to use the same tokenizer; UAG attempts to extend this to model combinations with different tokenizers. The article also provides a very practical insight: the assistant usually needs to be much smaller than the target to achieve meaningful latency benefits.

Suitable for: Low QPS, latency-sensitive scenarios with a very large target model and an available suitable small model. Not suitable for: Cases where the draft model itself is heavy, or the cost of cross-tokenizer alignment is too high.

3. EAGLE, MTP, MLP Speculator

The vLLM documentation lists various methods including EAGLE, MTP, draft model, PARD, MLP, N-gram, suffix, and dynamic speculative decoding, and provides guidance on method selection: model-based methods typically offer higher latency gains but also higher implementation and compatibility costs.

The common goal of these approaches is to reduce the serial cost of the drafting phase. For example, the Speculators project defines a speculator as a trainable draft model architecture deployable to vLLM, allowing a small drafter to predict multiple tokens in advance, which are then verified by the base model in one pass.

Suitable for: Teams with model training capabilities, clear inference cost targets, and a willingness to conduct stress tests and gradual rollouts.

4. Medusa / Tree-based Speculation

Medusa’s approach is to add multiple heads to the model that predict future tokens, forming multiple candidate paths. The TensorRT-LLM documentation emphasizes that Medusa’s potential comes from multi-path verification, but the width and depth of the tree need pruning; otherwise, the extra computation can offset the gains.

The SpecInfer paper further organizes candidate tokens into a token tree, using the target LLM as a verifier to verify candidate sequences in the tree in parallel, aiming to reduce end-to-end latency and computational cost.

Tree-based speculation offers greater potential gains but also higher engineering complexity. It requires handling candidate paths, sparse attention masks, acceptance path selection, memory overhead, and scheduling strategies.

Approach	Additional Model	Deployment Complexity	Suitable Scenarios	Gain Ceiling
N-gram / Prompt Lookup	None	Low	Summarization, Document QA, Code Editing	Medium
Draft Model	Requires small model	Medium	Low QPS, Large Model Service	High
EAGLE / MTP / MLP	Requires training	Medium-High	Teams with training capabilities	High
Medusa / Tree-based	Requires model modification	High	Deep optimization scenarios	Highest

Why Gains Disappear Under High Load

Many teams get a seemingly contradictory result when first testing speculative decoding: it’s fast in single-request tests, fast under low traffic, but once stressed to the target production QPS, the gains diminish significantly, or it even becomes slower than normal decoding.

There are typically four reasons:

First, drafting is not free. The draft model also consumes GPU, CPU, memory, KV Cache, and scheduling resources. Under low load, these costs might be offset by fewer target model steps; under high load, they enter the queuing system and become real overhead.

Second, continuous batching changes the baseline. Without speculative decoding, high QPS might already achieve high target model throughput through batching. The draft/verify phased flow introduced by speculative decoding can disrupt the originally simple and stable batching rhythm.

Third, the acceptance rate varies with the task. Code completion, document QA, and summarization usually have high local predictability; open-ended chat, long-chain reasoning, strong sampling, and multi-tool calls see a lower acceptance rate. As the acceptance rate drops, the deeper the speculation, the greater the waste.

Fourth, sampling parameters change the gains. Parameters like temperature, top_p, and top_k affect the target model’s distribution and the probability of draft tokens being accepted. Gains verified only under greedy decoding cannot be directly transferred to production sampling configurations.

A More Stable Deployment Path

Step 1: Establish a Baseline with Normal Decoding First

Don’t start by comparing “on” and “off.” First, establish a clear baseline with normal decoding.

At a minimum, record these metrics:

• TTFT p50 / p95 / p99
• ITL p50 / p95 / p99
• Output token throughput
• Request queue wait time
• GPU SM utilization and memory usage
• Average input and output length
• Latency distribution broken down by task type

Speculative decoding primarily improves the decode phase. Don’t mistakenly attribute failures to speculative decoding for long-context requests where prefill is the heavy part. Long-context services often require prefix caching, KV cache management, P/D disaggregation, and other techniques.

Step 2: Start with Low-Risk Methods

If your business involves RAG, document QA, code editing, or multi-turn context reuse, start with n-gram or suffix. They don’t require training a draft model and are suitable for the first round of online gradual rollout.

vLLM’s n-gram configuration can start with conservative parameters:

vllm serve Qwen/Qwen3-32B \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

The goal of this configuration is not to pursue extreme speedup, but to verify three things: whether the task has enough repeated segments, whether the acceptance rate is stable, and whether tail latency worsens.

Step 3: Then Evaluate Model-Based Speculators

If n-gram gains are limited but the business is still decode-heavy, evaluate draft models, EAGLE, MTP, or MLP speculators.

A typical draft model configuration looks like this:

vllm serve large-target-model \
  --speculative-config '{
    "method": "draft_model",
    "model": "small-draft-model",
    "num_speculative_tokens": 5,
    "draft_tensor_parallel_size": 1
  }'

Don’t just look at average speed. More importantly, observe: whether the draft model itself becomes a bottleneck, whether draft and target compete for the same resources, and whether the acceptance rate fluctuates significantly with prompt type.

Step 4: Implement Load-Based Routing, Not a Global Switch

A production system should not have a single boolean switch. A more reasonable approach is to enable different strategies based on traffic and request type.

# simplified routing logic
def choose_decode_policy(request, metrics):
    if metrics.qps_high or metrics.queue_wait_p95_ms > 200:
        return "baseline_decode"
    if request.task in ["document_qa", "summarization", "code_edit"]:
        return "ngram_speculation"
    if request.output_len_estimate > 256 and metrics.acceptance_rate_p50 > 0.65:
        return "draft_model_speculation"
    return "baseline_decode"

The core of this logic is: when the system is under high load, first protect stable throughput and tail latency; when the task is highly predictable, then enable more aggressive speculation depth.

Suitable Scenarios

Speculative decoding is more suitable for the following types of tasks:

• Code Completion and Code Editing: Code has strong structure, strong local patterns, and many repeated symbols, making draft tokens easier to accept.
• Document QA and Summarization: Output often rephrases entities, phrases, and structures from the input material; n-gram/prompt lookup can provide low-cost gains.
• Low QPS, High Interaction Chat Services: When the GPU is not fully saturated, the serial nature of decode is more apparent, and speculative decoding can improve perceived user speed.
• Large Model Services within a Fixed Model Family: If you can find a small model from the same family, an MTP checkpoint, or a trained speculator, the engineering integration cost will be lower.

Scenarios that are less suitable include: high QPS fully loaded services, strong random sampling, tasks with very low acceptance rates, large differences between the draft and target models, and long-context requests where the main bottleneck is prefill, not decode.

Common Misconceptions

Misconception 1: Only looking at tokens/s. Increased throughput does not equal improved user experience. Online services should focus more on TTFT, ITL, tail latency, and failure rates. If tokens/s increase but p99 ITL worsens, the user experience may be worse.

Misconception 2: Longer drafts are always better. num_speculative_tokens is not always better. As speculation depth increases, the costs of drafting, verification, KV management, and rejection waste also increase. The reasonable approach is to adjust dynamically based on acceptance rate and load.

Misconception 3: Smaller draft models are always better. A very small draft model will be fast but inaccurate; a large one can be accurate but not fast enough. A good drafter should balance being “fast enough” and “similar enough to the target model.”

Misconception 4: Offline benchmarks directly represent online performance. Offline tests with fixed batch sizes, input lengths, and output lengths cannot cover real online traffic. Online request length, task type, sampling parameters, and queuing status all vary.

Misconception 5: Ignoring the fallback mechanism. Speculative decoding must be able to quickly fall back to normal decoding. Without a fallback strategy, it’s difficult to handle sudden drops in acceptance rate, draft model anomalies, increased memory pressure, or longer queue waits.

Pre-Deployment Checklist

Model and Algorithm

• Confirm compatibility of target model, draft model, tokenizer, and sampling parameters.
• Confirm that the strict verification path does not change output quality.
• Test separately for greedy, temperature, top_p, and other sampling configurations.
• Record acceptance rates separately for different task types.

Performance and Capacity

• Compare normal decoding, n-gram, draft model, EAGLE/MTP, and other approaches.
• Test at low, medium, and high QPS.
• Record TTFT, ITL, output throughput, GPU utilization, memory, and queue wait times.
• Observe p95/p99 separately, not just averages.

Gradual Rollout and Fallback

• Roll out by task type, not globally.
• Set a minimum acceptance rate; automatically fall back if it drops below the threshold.
• Set a queue wait threshold; automatically fall back under high load.
• Keep a normal decode routing path.
• Monitor draft model errors, OOM, timeouts, and verification failures.

Cost Accounting

• Calculate the additional GPU cost of the draft model.
• Calculate the reduction in forward steps for the target model.
• Calculate the overall request cost, not just single-request speed.
• If using a remote drafter, additionally calculate network latency and cross-service scheduling overhead.

Summary

Speculative decoding is not a magic switch that makes things faster when turned on. It is a set of latency optimization techniques that require careful evaluation of traffic characteristics, task types, acceptance rates, and resource constraints. It can significantly improve ITL under low QPS, but gains may decay or even backfire under high QPS. A pragmatic path is to start with low-risk methods like n-gram, establish a baseline, implement layered routing, and always keep a fallback path — allowing your production system to find the true balance between “fast” and “stable.”

References

1. vLLM Documentation: Speculative Decoding (Accessed 2026-06-29)
2. NVIDIA TensorRT-LLM Documentation: Speculative Sampling / NGram / Medusa (Accessed 2026-06-29)
3. Hugging Face Blog: Universal Assisted Generation: Faster Decoding with Any Assistant Model (2024-10-29)
4. Hugging Face Blog: Faster Assisted Generation with Dynamic Speculation (2024-10-08)
5. vLLM Project: Speculators GitHub Repository (Accessed 2026-06-29)
6. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification (arXiv:2305.09781, v4 2024-04-01)
7. An Interpretable Latency Model for Speculative Decoding in LLM Serving (arXiv:2605.15051, Submitted 2026-05-14)
8. SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference (arXiv:2605.08151, Revised 2026-05-12)