Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

Background: Inference Optimization Shifts from ‘Faster per Token’ to ‘Phase Isolation’

In recent years, the keywords for LLM inference optimization have revolved around continuous batching, PagedAttention, quantization, KV cache reuse, and Speculative Decoding. These techniques focus on throughput, memory, and token generation efficiency within a single inference engine.

However, as workloads move into long contexts, RAG, codebase Q&A, multi-turn agents, and batch document processing, a new tension emerges: when the same pool of GPUs simultaneously handles prompt prefill and token-by-token generation, the two phases interfere with each other.

Online LLM inference naturally comprises two phases:

Phase	Characteristics	Bottleneck
Prefill	Computes attention states for all input tokens at once, generating KV cache	Compute-bound; heavier with longer inputs
Decode	Generates tokens one by one using the existing KV cache	Memory bandwidth and scheduling stability

When both phases share the same GPU pool, a new long prompt entering the system can preempt or batch-mix with ongoing decode requests. Hao AI Lab’s retrospective on DistServe summarizes this as prefill-decode interference: a new prefill either pauses decode or batches with it, both of which increase decode latency. The vLLM docs also clearly state that the goal of disaggregated prefilling is to separately optimize TTFT and ITL, and to control tail ITL, not to simply claim throughput improvement.

Core Problem: TTFT and TPOT Are Not the Same Optimization Target

Online LLM services have two key user experience metrics:

• TTFT (Time To First Token): The time from a user’s request to seeing the first token.
• TPOT / ITL (Time Per Output Token / Inter-Token Latency): The interval between each generated token.

Their engineering implications are fundamentally different:

Metric	Sensitive Factors
TTFT	Input length, prefill compute capacity, queue wait time, prefix cache hit rate
TPOT/ITL	Decode batch size, KV cache read/write, memory bandwidth, scheduling jitter

When placed in the same resource pool, the scheduler is forced to compromise—prioritizing lower TTFT can increase ITL for decoding requests, while protecting decode can cause new requests to queue, lengthening TTFT.

Prefill-Decode disaggregation is the core idea: don’t let these two different workloads compete for the same resource pool. Assign prefill and decode to separate instances, separate GPU pools, or even different hardware form factors, connecting the two phases via KV cache transfer.

This is not as simple as “just spinning up more services.” The real challenges are:

1. The KV cache generated by prefill must be reliably and quickly transferred to decode.
2. The request scheduler must know which decode worker can access the corresponding KV.
3. The prefill/decode resource ratio must be dynamically planned based on input/output lengths and SLOs.
4. The transfer link must not become a new bottleneck.

Core Principle: Splitting a Single Request into Two Independently Scalable Phases

A typical P/D disaggregated request flow:

1. API Gateway / Router receives request
         ↓
2. Scheduler routes based on input length, SLO, cache hit → sends to Prefill Worker
         ↓
3. Prefill Worker computes hidden states and KV cache
         ↓
4. KV Connector transfers KV cache to Decode Worker
         ↓
5. Decode Worker reads KV cache → generates tokens one by one
         ↓
6. Router streams the result back to the user

vLLM’s disaggregated prefilling provides a clear engineering model: run two vLLM instances (one prefill instance, one decode instance), using a Connector, LookupBuffer, and Pipe for KV cache transfer and matching. This essentially upgrades LLM serving from a single inference engine to a small distributed system—the benefit comes from isolation, and the cost comes from distribution.

Why This Path Matters in 2025–2026

After 2025, P/D disaggregation is no longer just an architectural concept in academic papers. Hao AI Lab’s November 2025 retrospective noted that almost all production-grade LLM serving frameworks are embracing some form of disaggregation, including NVIDIA Dynamo, SGLang, vLLM, LMCache, and Mooncake.

Research in 2026 further pushes the problem toward resource planning and cross-cluster forms:

• SLO-Aware Compute Resource Allocation formalizes the problem: given constraints on total throughput, SLOs, and input/output length distributions, how to determine the number of resources for prefill and decode.
• Large-Scale LLM Inference with Heterogeneous Workloads discusses prefill/decode contention under heterogeneous workloads from a queuing network and scheduling control perspective.
• Prefill-as-a-Service explores offloading long-context prefill to a separate cluster and transferring the KV cache back to a local decode cluster.

These works collectively point to a trend: the unit of LLM inference optimization is moving from kernels, single GPUs, and single instances to resource pools, networks, and scheduling policies.

Engineering Implementation: Evolving from Single Instance to P/D Disaggregation

Step 1: Separate Your Metrics First

Don’t change the architecture right away. First, split your existing service’s observability into these groups:

• TTFT: Bucketed by input length (0–1K, 1K–4K, 4K–16K, 16K+)
• TPOT/ITL: Bucketed by output length and concurrency
• p95/p99 Tail Latency: Monitor separately, not just averages
• Queue Time: Record prefill queue time and decode queue time separately
• Resource Metrics: GPU utilization, memory bandwidth, KV cache usage and hit rate

Only when you confirm that “long prompts or bursty prefill is increasing decode tail latency” does P/D disaggregation have a clear benefit target.

Step 2: Logical Decoupling First, Physical Decoupling Later

Early on, avoid cross-node KV transfer. It’s recommended to first separate prefill/decode workers within the same node and high-speed interconnect domain to reduce engineering variables:

1. Single-instance continuous batching
2. Chunked prefill to mitigate the impact of long prompts on decode
3. Same-node P/D separation, using high-bandwidth links for KV transfer
4. Multi-node P/D separation, introducing a KV-aware router
5. KV cache storage layer or multi-level cache, supporting cross-instance reuse
6. Heterogeneous resource pool: compute-oriented resources for prefill, bandwidth-oriented resources for decode

vLLM docs note: chunked prefill with an appropriate chunk size can achieve similar control over tail ITL, but tuning is not easy; disaggregated prefilling is more reliable but marked as experimental and does not directly improve throughput.

Step 3: Treat KV Cache as a First-Class Citizen

The key asset in P/D disaggregation is not the request body, but the KV Cache. Once split, the KV cache transitions from an in-process state to a data plane that is transferred across workers. Four things must be designed:

Dimension	Key Question
KV Identity	How to name, look up, and expire KV for a request/prefix/batch
KV Routing	How does a decode worker know where to read the KV from
KV Transfer	Via NVLink, InfiniBand, PCIe, Ethernet, or a hybrid hierarchy
KV Lifecycle	When to release, how to clean up after failure, whether to allow multi-request reuse

LMCache exposes the KV cache at the LLM engine interface layer, transforming the engine from an “independent token processor” into a “collection of engines with KV cache as the storage and communication medium.” This is especially important for enterprise deployments, where prompt prefixes are often highly repetitive (system prompts, tool descriptions, document templates, agent trajectory prefixes).

Step 4: Don’t Guess Resource Ratios

The most common problem after P/D disaggregation: how many GPUs for the prefill pool and how many for the decode pool?

The answer depends on the business traffic distribution. A simplified estimation model:

prefill_load ≈ request_qps × avg_input_tokens / prefill_tokens_per_second
decode_load  ≈ request_qps × avg_output_tokens / decode_tokens_per_second

Then correct using p95 input length, p95 output length, and SLO constraints. The length distribution for long-context workloads is typically heavy-tailed; a few very long requests can determine tail latency performance. The 2026 SLO-Aware P/D resource allocation research also follows a path of “theoretical modeling + empirical benchmarking”: first estimate, then correct using SLO constraints like TTFT and TPOT.

Applicable Scenarios

P/D disaggregation is more suitable for the following types of workloads:

1. Long-Context RAG: Retrieved chunks + system prompts + user questions can result in prompts of thousands to tens of thousands of tokens; prefill costs are high and volatile.
2. Codebase Q&A and Code Generation: Long inputs and long outputs coexist; both TTFT and TPOT need attention.
3. Multi-Turn Agents: Carries system prompts, tool descriptions, historical trajectories, and intermediate results; prefix cache and append-prefill both influence architecture choices.
4. Multi-Tenant Inference Platforms: Different tenants have vastly different input/output lengths, SLOs, and pricing models; P/D disaggregation supports finer-grained resource isolation and elastic scaling.
5. Large-Scale GPU Clusters: With hundreds or even thousands of GPUs, a unified resource pool amplifies scheduling interference; P/D disaggregation allows independent scaling per phase.

Scenarios Where It’s Not Suitable or Requires Caution

• Workloads with short inputs, short outputs, and low QPS: Single-instance continuous batching + PagedAttention + prefix cache may be sufficient.
• Scenarios with poor network conditions: KV cache transfer will become a bottleneck.
• Teams lacking basic observability: Without splitting TTFT, TPOT, queue times, and KV transfer time, it’s impossible to identify the root cause of problems.

Common Misconceptions

Misconception	Truth
P/D disaggregation always improves throughput	vLLM docs explicitly state it does not improve throughput; the core value is tail latency governance.
Splitting into two services is P/D disaggregation	Must handle KV cache transfer, matching, lifecycle, and failure recovery.
Plan resources based only on average lengths	For long-context workloads, look at p90/p95/p99; averages mask the long tail.
Ignoring multi-turn append-prefill	Not every turn requires a full prefill; PPD Disaggregation research notes that some turn 2+ can be handled locally on the decode node.

Pre-Launch Checklist

Metrics and Stress Testing

• Split TTFT by input length, output length, tenant, and model type.
• Monitor TPOT/ITL p50/p95/p99 separately.
• Record prefill queue time, decode queue time, and KV transfer time.
• Cover mixed stress tests with bursty traffic, long contexts, short Q&A, and multi-turn agents.
• Perform resource ratio sweeps (e.g., 1:1, 1:2, 1:3).

KV Cache

• KV keys are traceable, with TTL and cleanup mechanisms.
• KV does not leak when prefill succeeds but decode fails.
• Decode workers can re-fetch or discard corresponding KV after restart.
• Retry/degrade/failover strategies for KV transfer failures.

Scheduling and Routing

• Scheduler knows the request’s input length, estimated output length, and SLO.
• Router is KV-aware.
• Long-prompt admission control and tenant-level rate limiting are in place.
• Avoid decode workers being overwhelmed by a sudden peak of prefill KV.

Release and Rollback

• Support canary deployment to a subset of models or tenants.
• Retain the original co-located serving rollback path.
• Dashboard can compare TTFT, TPOT, and GPU utilization before and after decoupling.

A Reference Architecture for Implementation

Client / SDK → API Gateway → SLO-aware Router
                                  ↓
                Prefill Pool ← → KV Transfer / KV Store ← → Decode Pool
                                                              ↓
                                                     Streaming Response
                                       ↑
                                  Observability

Key module responsibilities:

• API Gateway: Authentication, tenant identification, request size limits, streaming response management.
• SLO-aware Router: Routes based on input length, tenant tier, model, and cache hit.
• Prefill Pool: Compute-intensive; can use parallel strategies suitable for prompt processing.
• KV Transfer / KV Store: Responsible for KV transfer, caching, reuse, expiration, and cleanup.
• Decode Pool: Focuses on stable generation and bandwidth utilization; prioritizes protecting TPOT/ITL.
• Observability: Full-trace tracking of TTFT, TPOT, KV transfer, queues, and failure rates.

For small to medium-sized teams, the first version should prioritize a minimal closed loop within the same data center and high-speed network domain. Validate tail latency benefits before gradually introducing multi-level KV caching and heterogeneous resource pools.

Conclusion: P/D Disaggregation is a Latency Governance Architecture, Not a Single-Point Performance Switch

The value of Prefill-Decode disaggregation is not in making the service more complex, but in acknowledging that the two phases of LLM inference have different resource profiles and different user experience metrics.

When workloads enter the stages of long contexts, multi-turn agents, multi-tenancy, and high concurrency, a single GPU pool can easily create trade-offs between TTFT, TPOT, throughput, and cost. P/D disaggregation provides a more controllable architecture: let prefill focus on processing context, let decode focus on stable generation, and let KV cache become the first-class data plane connecting them.

However, it also introduces new engineering costs. Before launch, three questions must be answered:

1. Is your tail latency problem truly caused by prefill/decode interference?
2. Is your KV cache transfer link fast enough, observable, and recoverable?
3. Are your resource ratios based on real input/output lengths and SLO stress testing, not just educated guesses?

If the answers are yes, P/D disaggregation will become a key architecture for moving long-context LLM serving from “working” to “stable, scalable, and cost-controllable.”

Key References

1. vLLM Documentation, Disaggregated Prefilling: https://docs.vllm.ai/en/latest/features/disagg_prefill/
2. Hao AI Lab, Disaggregated Inference: 18 Months Later: https://haoailab.com/blogs/distserve-retro/
3. Li et al., SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference: https://arxiv.org/abs/2603.04716
4. Lin et al., Large-Scale LLM Inference with Heterogeneous Workloads: https://arxiv.org/abs/2602.02987
5. Zhang et al., SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference: https://arxiv.org/abs/2510.08544
6. Cheng et al., LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference: https://arxiv.org/abs/2510.09665
7. Qin et al., Prefill-as-a-Service: https://arxiv.org/abs/2604.15039
8. Li et al., Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving: https://arxiv.org/abs/2603.13358