Prefill-Decode Disaggregation in Production: Separating Prefill and Decode to Reduce TTFT/TPOT Conflicts

Background: LLM Inference is Not a Uniform Load

When many teams first deploy a large language model service, they often think of inference as “a request occupies a GPU, and the model generates tokens step by step.” This is an oversimplification. For a decoder-only LLM, a single request is typically split into two phases: Prefill and Decode.

• Prefill processes the entire input prompt at once, computing the Key/Value states for each layer and generating the first output token. It is generally compute-intensive, especially in long-context scenarios like RAG, multi-turn conversations, and code repository analysis, where input tokens are long and matrix computations are heavy.
• Decode generates subsequent tokens auto-regressively, one at a time, based on the existing KV Cache. It is more sensitive to memory bandwidth, KV cache reads, and batch scheduling. Each step has relatively low computation, but requires consistent, stable, low-jitter execution.

The problem arises from mixed production traffic: some requests have long inputs, some have long outputs, some need only a few dozen tokens, and others stream thousands of tokens. If Prefill and Decode are always scheduled on the same set of GPUs, a long prompt’s Prefill can interrupt an ongoing Decode, causing the user to experience a sudden slowdown in subsequent tokens after streaming has already started. The result is: TTFT seems acceptable, but TPOT or ITL tail latency degrades; or, to protect the decode experience, prefill is throttled, leading to longer first-token wait times for new requests.

Prefill-Decode Disaggregation aims to separate these two phases with different load characteristics, allowing the system to optimize TTFT (Time To First Token) and TPOT (Time Per Output Token) independently, rather than having them compete for resources on the same GPUs, parallel strategy, and scheduling queue.

Core Principle: Separate “Reading the Input” and “Continuous Generation” into Two Resource Pools

Traditional Aggregated Inference

In aggregated serving, a request is typically processed within the same set of model instances:

Client Request → Router → Model Worker: Prefill → Same Model Worker: Decode → Streaming Response

This approach is simple, has low debugging overhead, and is sufficient for single-node deployments and low-to-medium traffic. However, it binds the two types of workloads together:

• Prefill and Decode share the same GPUs.
• Prefill and Decode use the same tensor parallel / pipeline parallel configuration.
• Long prompts can affect requests that are already streaming.
• Scaling only works by adding complete model replicas, making it difficult to independently increase Prefill or Decode capacity.

Disaggregated Inference

In disaggregated serving, the system splits a request into two stages:

Client Request → Router → Prefill Queue → Prefill Worker Pool → KV Cache Transfer → Decode Worker Pool → Streaming Response

The Prefill Worker processes the input prompt and generates the KV Cache needed for subsequent decoding. The system then transfers this KV Cache to a Decode Worker via high-speed networking, shared cache, KV connector, or a dedicated KV transfer runtime. The Decode Worker does not reprocess the full input; it receives the existing state and continues generation.

This yields three engineering benefits:

1. Resource allocation can be adjusted independently: Increase Prefill resources for long-context requests; increase Decode resources for long-output requests.
2. Parallel strategies can be optimized per phase: Prefill favors compute throughput; Decode favors memory bandwidth and low jitter.
3. Tail latency is more controllable: Long Prefill operations do not frequently interrupt Decode, resulting in more stable token intervals for streaming output.

However, it introduces a new cost: KV Cache transfer. When prompts are short, the Decode side’s prefix cache hit rate is high, or cross-node bandwidth is insufficient, remote Prefill may be counterproductive. Therefore, the key to production deployment is not to disaggregate all requests, but to dynamically decide whether to disaggregate based on request characteristics.

Key Mechanism 1: Decide Between Remote and Local Prefill Per Request

A mature disaggregated system typically does not send all requests to a remote Prefill Pool. A better approach is to design a Disaggregated Router that makes decisions based on the following factors:

Decision Factor	Description
Input token count	Long inputs are better suited for remote Prefill
Estimated output length	Long outputs require protecting the Decode Pool’s continuous generation
Prefill Queue wait time	When the queue is long, short requests may be faster with local processing
Decode Worker’s prefix cache hit rate	If the local cache is already rich, remote Prefill may cause redundant transfers
Tenant priority and SLA	Interactive, batch, and background summarization tasks should have different strategies

A simplified rule can be expressed as:

pd_disaggregation_policy:
  remote_prefill:
    min_input_tokens: 2048
    max_prefill_queue_wait_ms: 300
    min_expected_output_tokens: 128
    require_decode_pool_pressure: true
  local_prefill:
    max_input_tokens: 1024
    prefer_when_prefix_cache_hit: true
fallback:
  on_kv_transfer_timeout: local_aggregated_serving
  on_prefill_queue_overload: local_prefill

This configuration is not a recommended default; it illustrates the policy structure. Before going live, you must test with your specific model size, GPU type, network bandwidth, context length distribution, and business SLA.

Key Mechanism 2: KV Cache Transfer Determines if Disaggregation is Worthwhile

Prefill-Decode disaggregation is not a free lunch. It requires the system to transfer the KV Cache generated during the Prefill phase from the Prefill Worker to the Decode Worker. Several details are often underestimated.

Transfer Volume Grows with Context and Model Size

KV Cache size depends on the number of layers, hidden size, attention head configuration, sequence length, and precision. The longer the context, the larger the transfer volume. For long-context models, KV transfer can become a new bottleneck.

KV Layout is Not Necessarily Compatible

Different inference engines, attention backends, paged KV management methods, and tensor parallel configurations may use different layouts. When transferring across workers, layout transformation may be required. If this step is not asynchronous or overlapped with computation, it will directly increase TTFT.

Transfer Must be Observable, Timeout-able, and Fallback-able

In a production system, you cannot just measure model forward time. You must at least break down the following phases:

request_queue_wait
prefill_compute_time
kv_cache_pack_time
kv_cache_transfer_time
kv_cache_unpack_time
decode_queue_wait
first_token_emit_time
per_token_decode_time

Otherwise, when users report “first token is slow” or “streaming output stutters,” it is difficult to determine whether the cause is Prefill queuing, network transfer, Decode congestion, or a drop in cache hit rate.

Key Mechanism 3: Capacity Planning Must Consider Both TTFT and TPOT

Typical online services use QPS, average latency, and p95 latency for capacity planning. LLM serving requires a more granular set of metrics:

Metric	Description
TTFT	Time from user request to first token
TPOT / ITL	Average or percentile time for each subsequent token
Goodput	Effective request rate that meets both TTFT and TPOT SLOs
Prefill Queue Delay	Time requests spend waiting in the Prefill queue
Decode Active Sequences	Number of sequences currently being generated in the Decode Pool
KV Transfer Time	End-to-end transfer time of KV Cache from Prefill to Decode
Cache Hit Rate	Hit rate for prefix cache or shared KV cache

A more practical capacity goal is not “maximize GPU utilization,” but: maximize goodput while meeting TTFT p95, TPOT p95, error rate, and cost budget.

If you only pursue GPU utilization, the system might increase surface-level utilization by packing in more Prefill tasks, but at the expense of Decode smoothness. Users will not see “GPU is busier”; they will see output stuttering.

Engineering Implementation: An Actionable Migration Path

Phase 1: Establish a Baseline Profile

Before making changes, collect a 3-7 day profile of real traffic:

• Input token length distribution
• Output token length distribution
• Proportion of RAG vs. non-RAG requests
• Multi-turn conversation context length
• Percentage of users using streaming output
• Current TTFT, TPOT, E2E latency, GPU utilization
• Batch size, active sequences, KV cache usage

If 90% of requests are short-input and short-output, Prefill-Decode disaggregation is likely not a priority. In that case, focus on continuous batching, prefix caching, chunked prefill, quantization, or prompt compression.

Phase 2: Introduce Remote Prefill with Local Fallback

For the first version, do not switch all traffic. Start with a canary based on model, tenant, endpoint, or input length:

rollout:
  enabled_models:
    - qwen-long-context-service
    - rag-answer-service
  traffic_percent: 5
  remote_prefill_when:
    input_tokens_gte: 4096
  stream_response: true
  fallback_to_aggregated: true

Always keep the local aggregated path. If the remote Prefill queue, KV transfer, or Decode reception encounters an anomaly, you can directly fall back to the original serving mode.

Phase 3: Independent Scaling

One of the values of a disaggregated architecture is the ability to scale the Prefill Pool and Decode Pool independently. Do not base scaling decisions solely on GPU utilization; also consider phase-level queues:

• Long Prefill Queue, rising TTFT: Add Prefill Workers or relax local Prefill conditions.
• High Decode Active Sequences, rising TPOT: Add Decode Workers or limit long-output tasks.
• Rising KV Transfer Time: Check network, layout transformation, transfer concurrency, and caching strategy.
• Falling Cache Hit Rate: Check if routing is breaking prefix locality.

Phase 4: Multi-Tenant Isolation

Different business types have different sensitivities to TTFT and TPOT:

Business Type	Latency Sensitivity	Optimization Focus
Chatbot	TTFT and streaming TPOT	Low latency, stable streaming
Batch Summarization	Throughput and cost	High throughput, low cost
Agent Tool Calling	Short response, stable latency	Low jitter
RAG Long-Context QA	Prefill capacity	High throughput for large contexts

Therefore, do not let all businesses share the same disaggregation strategy. At a minimum, support configuration by tenant, route, model, and prompt length bucket.

Applicable Scenarios

Scenarios suitable for Prefill-Decode Disaggregation:

1. Long-context RAG: Many input documents, long prompts, significant Prefill pressure.
2. Code repository Q&A: Large context, first-token latency prone to fluctuation.
3. Enterprise knowledge base assistant: Multi-tenant, mixed long and short requests, need to protect interactive experience.
4. High-concurrency streaming chat: Decode phase is continuously occupied and should not be frequently interrupted by large Prefill operations.
5. Heterogeneous GPU clusters: Use compute-optimized GPUs for Prefill and memory-bandwidth-optimized GPUs for Decode.

Scenarios less suitable:

• Single GPU or small-scale deployments.
• Low-concurrency services with short inputs and outputs.
• Environments with weak network bandwidth and high cross-node transfer costs.
• Early-stage systems lacking phase-level monitoring and fallback mechanisms.

Common Misconceptions

Misconception 1: Disaggregation Always Increases Throughput

Not necessarily. As the vLLM documentation notes, the primary value of disaggregated prefilling is to independently tune TTFT and ITL and control tail ITL, not to unconditionally boost throughput. When the proportion of short requests is high, the overhead of remote Prefill queuing and KV transfer may negate the benefits.

Misconception 2: Fast Network is Sufficient

Bandwidth is only one dimension. KV Cache also involves packing, unpacking, layout transformation, memory allocation, synchronization points, error fallback, and cross-worker state management. Any of these steps can become a bottleneck and pollute TTFT.

Misconception 3: Only Monitor GPU Utilization

High GPU utilization does not mean a good user experience. Too many Prefill tasks can increase compute utilization but sacrifice Decode stability. When launching, put TTFT, TPOT, queue wait times, transfer times, and error rates on the same dashboard.

Misconception 4: All Requests Should Use Remote Prefill

The correct approach is conditional disaggregation. Local Prefill may be better for short inputs, when the local prefix cache is hit, or when the Prefill queue is congested.

Misconception 5: Disaggregation is Just an Inference Engine Parameter

It is not a simple on/off switch. It is a system engineering effort involving scheduling, caching, networking, parallel strategies, monitoring, fallback, and capacity planning.

Production Launch Checklist

Workload Profiling

• Input token length distribution has been collected.
• Output token length distribution has been collected.
• Business types (RAG, Chat, Agent, Batch) have been distinguished.
• p95/p99 SLOs for TTFT and TPOT have been defined.

Routing Strategy

• Conditions for remote vs. local Prefill have been defined.
• Prefix cache hit rate has been considered.
• Fallback strategy for Prefill Queue overload has been configured.
• Canary support by model, tenant, and endpoint is in place.

KV Cache Transfer

• KV packing, transfer, and unpacking times are being monitored.
• KV layout compatibility across different parallel configurations has been verified.
• Transfer timeouts and failure fallbacks have been set.
• Network bandwidth and cross-node topology have been evaluated.

Scheduling and Capacity

• Prefill Pool and Decode Pool have been stress-tested independently.
• Prefill Queue growth under burst traffic has been verified.
• Maximum active sequence limit for the Decode Pool has been configured.
• SLO-based scaling rules have been established.

Observability and Rollback

• TTFT, TPOT, and E2E latency percentile dashboards have been created.
• Each request’s prefill/decode worker, routing decision, and transfer time are logged.
• Alerts are configured for: transfer timeouts, queue backlogs, TPOT jitter, and cache hit rate drops.
• One-click rollback to aggregated serving is supported.

Summary

Prefill-Decode Disaggregation is not a silver bullet, but it provides a clear, layered optimization path for LLM serving scenarios with long contexts, high concurrency, and mixed traffic. The core lies in three points: dynamic routing based on request characteristics, controlling KV cache transfer costs, and building an observable capacity system around TTFT and TPOT. If your team is facing bottlenecks from Prefill and Decode interfering with each other, start with traffic profiling and canary experiments to gradually verify if this architecture is suitable for your production environment.

References

1. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — arXiv:2401.09670
2. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — arXiv:2311.18677
3. vLLM Documentation: Disaggregated Prefilling — docs.vllm.ai
4. TensorRT-LLM: Disaggregated Serving in TensorRT LLM — NVIDIA Tech Blog
5. NVIDIA Dynamo Documentation: Disaggregated Serving — docs.nvidia.com
6. NVIDIA Dynamo GitHub Repository — github.com/ai-dynamo/dynamo
7. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — arXiv:2403.02310
8. NVIDIA Technical Blog: Removing the Guesswork from Disaggregated Serving — developer.nvidia.com