Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

Background: Inference Optimization Has Shifted from “Faster Single Runs” to “Phase Isolation”

In the past, discussions about LLM inference optimization revolved around keywords like Continuous Batching, PagedAttention, Quantization, KV Cache Reuse, and Speculative Decoding. These techniques addressed throughput, memory, and token generation efficiency within a single inference engine.

However, as applications move into long contexts, RAG, codebase Q&A, multi-turn agents, and batch document processing, a more challenging problem emerges: When the same pool of GPUs simultaneously handles prompt prefill and token-by-token generation, the two phases interfere with each other.

Online LLM inference is typically split into two phases:

Phase	Characteristic	Bottleneck
Prefill	Computes attention states for all input tokens at once, generating the KV cache	Compute-bound, heavier with longer inputs
Decode	Generates tokens one by one using the existing KV cache	Memory bandwidth and scheduling stability

When these two phases share the same GPU pool, a new long prompt can preempt resources from ongoing decode requests, causing visible stuttering in the user’s output. Hao AI Lab’s retrospective on DistServe summarizes this as Prefill-Decode Interference: when prefill and decode share GPUs, a new prefill either pauses decode or batches with it, both of which increase decode latency. The vLLM documentation also explicitly states that the goal of disaggregated prefilling is to separately optimize TTFT and ITL and control tail ITL, not simply to claim throughput improvements.

Core Problem: TTFT and TPOT Are Not the Same Optimization Target

Online LLM services have at least two user-perceivable metrics:

• TTFT (Time To First Token): The time a user waits to see the first token after sending a request.
• TPOT / ITL (Time Per Output Token / Inter-Token Latency): The interval between each generated token.

These metrics have very different engineering implications:

Metric	Sensitive Factors
TTFT	Input length, prefill compute, queue wait, prefix cache hit rate
TPOT/ITL	Decode batch size, KV cache read/write, memory bandwidth, scheduling jitter

When they share the same resource pool, the scheduler can only make compromises: inserting a prefill to reduce TTFT can increase the ITL of ongoing decode requests; protecting decode can cause new requests to queue, worsening TTFT.

The basic idea behind Prefill-Decode Disaggregation is therefore straightforward: don’t let these two different workloads compete for the same resource pool. Assign prefill and decode to different instances, different GPU pools, or even different hardware types, and connect the two phases via KV cache transfer.

This is not simply “running more services.” The real challenges are:

1. The KV cache generated after prefill must be reliably and quickly transferred to the decode worker.
2. The request scheduler must know which decode worker has the corresponding KV cache.
3. The prefill/decode resource ratio must be dynamically planned based on input length, output length, and SLOs.
4. The transfer link must not become a new bottleneck.

Core Principle: Splitting a Single Request into Two Independently Scalable Phases

A typical P/D disaggregation flow looks like this:

1. API Gateway / Router receives the request.
2. Scheduler routes the request to a prefill worker based on input length, SLO, and cache hit status.
3. Prefill worker computes the hidden states and KV cache for the prompt.
4. KV Connector transfers the KV cache to a decode worker or writes it to a shared KV storage layer.
5. Decode worker reads the KV cache and begins token-by-token generation.
6. Router streams the generated output back to the user.

vLLM’s disaggregated prefilling documentation provides a clear engineering model: run two vLLM instances, one as a prefill instance and one as a decode instance, connected by a connector that transfers prefill KV caches and results. Its internal abstractions include Connector, LookupBuffer, and Pipe; the Connector allows the KV consumer to obtain the KV cache produced by the KV producer, LookupBuffer supports insert/drop_select semantics, and the Pipe handles tensor transfer.

From an architectural perspective, P/D disaggregation transforms LLM serving from “a single inference engine” into a small distributed system. Its benefits come from isolation, and its costs also come from distribution: network transfer, cache coherency, routing, failure recovery, and resource planning all become more complex.

Why This Path Became Important in 2025-2026

After 2025, P/D disaggregation is no longer just a service architecture in research papers. Hao AI Lab’s November 2025 retrospective noted that almost all production-grade LLM serving frameworks have begun to embrace some form of disaggregation:

• NVIDIA Dynamo
• llm-d
• Ray Serve LLM
• SGLang
• vLLM
• LMCache
• Mooncake

Long contexts, business scaling, and stricter tail-latency requirements are the main drivers of this shift.

Research in 2026 further pushed the problem toward resource planning and cross-cluster architectures. For example, SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference formulates the problem as: given constraints on total throughput, SLOs, and input/output length distributions, how many prefill and decode resources are needed? Large-Scale LLM Inference with Heterogeneous Workloads discusses prefill/decode contention under heterogeneous workloads from the perspective of queuing networks and scheduling control. A more radical direction is Prefill-as-a-Service, which selectively externalizes long-context prefill to an independent prefill cluster and transfers the KV cache back to a local decode cluster.

These works collectively point to a trend: The unit of LLM inference optimization is moving from kernels, single GPUs, and single instances to resource pools, networks, and scheduling policies.

Engineering Implementation: Evolving from Single-Instance Serving to P/D Disaggregation

Step 1: Separate Your Metrics First

Don’t change the architecture right away. First, break down your existing service metrics into the following groups:

• TTFT: Bucketed by input length (0–1K, 1K–4K, 4K–16K, 16K+).
• TPOT/ITL: Bucketed by output length and concurrency.
• p95/p99: Monitor tail latency separately, not just averages.
• Prefill queue time and decode queue time: Record wait times separately.
• GPU utilization, memory bandwidth, KV cache usage, KV cache hit rate.

Only when you confirm that “long prompts or bursty prefill are increasing decode tail latency” does P/D disaggregation have a clear benefit target.

Step 2: Start with Logical Disaggregation, Then Physical

Early on, you don’t need to transfer KV across nodes. You can separate prefill and decode workers on the same node or within the same high-speed interconnect domain to reduce engineering variables.

A recommended evolution sequence:

Phase	Approach
1	Single-instance continuous batching
2	Chunked prefill to mitigate the impact of long prompts on decode
3	Same-node P/D separation, using high-bandwidth links for KV transfer
4	Multi-node P/D separation, introducing a KV-aware router
5	KV cache storage layer or multi-level cache, supporting cross-instance reuse and more complex scheduling
6	Heterogeneous resource pools: compute-oriented resources for prefill, bandwidth-oriented resources for decode

The vLLM documentation also notes that chunked prefill can achieve similar goals for controlling tail ITL with the right chunk size, but finding the right chunk size is not easy; disaggregated prefilling is more reliable, but it is explicitly marked as experimental and does not directly improve throughput.

Step 3: Treat KV Cache as a First-Class Citizen

The key asset in P/D disaggregation is not the request body, but the KV Cache. Once you split the phases, the KV cache goes from being an in-process state to a data plane that is transferred between workers.

At a minimum, you need to design for four things:

Dimension	Key Questions
KV Identity	How to name, find, and expire the KV cache for a request, a prefix, or a batch
KV Routing	How does a decode worker know where to read the KV cache from
KV Transfer	Via NVLink, InfiniBand, PCIe, Ethernet, CPU memory, SSD, or a hybrid hierarchy
KV Lifecycle	When to release, how to clean up on failure, whether to allow reuse by multiple requests

LMCache’s design philosophy is worth considering: it exposes the KV cache at the LLM engine interface layer, transforming the engine from an “independent token processor” into a “collection of engines using KV cache as a storage and communication medium,” and supports cache offloading and P/D disaggregation. This direction is especially important for enterprise deployments, where prompt prefixes often have significant repetition—system prompts, tool descriptions, document templates, and agent trajectory prefixes all fall into this category.

Step 4: Don’t Guess Resource Ratios

The most common question after P/D disaggregation is: how many GPUs should be in the prefill pool and the decode pool?

The answer depends on the business traffic distribution:

Business Type	Input Characteristics	Output Characteristics
Document Q&A	Long input	Short output
Code Generation	Medium input	Long output
Multi-turn Agent	Multi-turn, shared context	Unstable tool call intervals

You can use a simplified model for initial estimation:

prefill_load ≈ request_qps × avg_input_tokens / prefill_tokens_per_second
decode_load  ≈ request_qps × avg_output_tokens / decode_tokens_per_second

Then correct with p95 input length, p95 output length, and SLO constraints. The 2026 SLO-Aware P/D resource allocation research also uses a “theoretical modeling + empirical benchmarking” approach: first estimate resources based on throughput requirements, input/output lengths, and prefill/decode throughput, then calibrate with SLO constraints like TTFT and TPOT.

Don’t just use averages in engineering. The length distribution in long-context applications is often heavy-tailed; a few very long requests can determine tail latency performance.

Applicable Scenarios

P/D disaggregation is better suited for the following types of applications:

1. Long-Context RAG — When input documents, retrieval snippets, system prompts, and user questions are concatenated, the prompt can easily reach thousands to tens of thousands of tokens. Prefill costs are high and volatile, easily interfering with ongoing generation.

2. Codebase Q&A and Code Generation — Code scenarios often have both long inputs and long outputs. Prefill needs to process a lot of context, and decode may generate long code blocks. Both TTFT and TPOT need attention.

3. Multi-turn Agents — Agents carry system prompts, tool descriptions, historical trajectories, and intermediate results. In multi-turn requests, prefix cache, append-prefill, and local decode reuse can all influence architecture choices.

4. Multi-tenant Inference Platforms — Different tenants have different input/output lengths, SLOs, and pricing models. P/D disaggregation allows for finer-grained resource isolation, routing policies, and elastic scaling.

5. Large-Scale GPU Clusters — When a cluster reaches hundreds or thousands of GPUs, a unified resource pool amplifies scheduling interference. P/D disaggregation shifts resource planning from “over-provisioning for the worst case” to “independent scaling by phase.”

Scenarios Where It’s Not Applicable or Requires Caution

P/D disaggregation is not a silver bullet.

• If your application has short inputs, short outputs, and low QPS, single-instance continuous batching + PagedAttention + prefix cache may be sufficient. In this case, the KV transfer, deployment complexity, and troubleshooting costs of P/D disaggregation may outweigh the benefits.
• If network conditions are weak, KV cache transfer can become a bottleneck. The KV cache for long-context dense attention models is large, and cross-node or cross-datacenter transfer may not be cost-effective.
• If your team lacks basic observability, it’s not recommended to jump straight to P/D. Without splitting TTFT, TPOT, queue times, KV transfer time, and tail latency, you won’t be able to determine if the problem is insufficient prefill compute, insufficient decode bandwidth, slow KV transfer, or a bad scheduling policy.

Common Misconceptions

Misconception 1: P/D Disaggregation Always Improves Throughput

The vLLM documentation explicitly states that disaggregated prefill does not improve throughput. Its core value is separately optimizing TTFT and ITL and controlling tail ITL. Whether throughput improves depends on whether resource over-provisioning is eliminated, phase interference is reduced, and KV transfer costs are low enough.

Misconception 2: Simply Running Two Services Is P/D Disaggregation

True P/D disaggregation must handle KV cache transfer, matching, lifecycle management, and failure recovery. Otherwise, you’ve just turned an in-process problem into a cross-service problem.

Misconception 3: Planning Resources Based Only on Average Input/Output Lengths

Averages mask the long tail. Long-context services must look at p90/p95/p99 input lengths, output lengths, and concurrency distributions.

Misconception 4: Ignoring Append-Prefill in Multi-Turn Requests

In multi-turn scenarios, not every turn requires a full prefill. Research on PPD Disaggregation in 2026 points out that the cost and interference of append-prefill in multi-turn requests are less than full prefill, so it may be beneficial to dynamically decide whether turn 2+ requests should be processed locally on the decode node. In engineering, a fixed “all prefill is externalized” strategy may not be optimal.

Go-Live Checklist

Metrics and Stress Testing

• Is TTFT split by input length, output length, tenant, and model type?
• Are TPOT/ITL p50/p95/p99 monitored separately?
• Are prefill queue time, decode queue time, and KV transfer time tracked?
• Is there a mixed stress test with bursty traffic, long contexts, short Q&A, and multi-turn agents?
• Has a resource ratio sweep been performed (e.g., 1:1, 1:2, 1:3 prefill/decode GPU ratios)?

KV Cache

• Is the KV key traceable?
• Does the KV cache have a TTL and cleanup mechanism?
• If prefill succeeds but decode fails, will the KV cache leak?
• After a decode worker restarts, can it re-fetch or abandon the corresponding KV cache?
• When KV transfer fails, does the request retry, degrade, or fail outright?

Scheduling and Routing

• Does the scheduler know the input length, estimated output length, and SLO of the request?
• Is the router KV-aware?
• Is there a mechanism to prevent decode workers from being overwhelmed by a sudden burst of prefill KV data?
• Are there tenant-level rate limits and long-prompt admission controls?

Release and Rollback

• Can you roll out the change to a subset of models or tenants?
• Is the original co-located serving path preserved for rollback?
• Can P/D disaggregation be disabled per model, per business, or per tenant?
• Is there a dashboard to compare TTFT, TPOT, and GPU utilization before and after disaggregation?

A Practical Reference Architecture

A P/D disaggregation system can be broken down into six layers:

Client / SDK
    ↓
API Gateway
    ↓
SLO-aware Router
    ↓
Prefill Pool  ←→  KV Transfer / KV Store  ←→  Decode Pool
                                                    ↓
                                            Streaming Response

Key module responsibilities:

Module	Responsibility
API Gateway	Authentication, tenant identification, request size limits, streaming response management
SLO-aware Router	Routes requests based on input length, tenant tier, model, and cache hit status
Prefill Pool	Compute-intensive; can use parallel strategies better suited for prompt processing
KV Transfer / KV Store	Handles KV transfer, caching, reuse, expiration, and cleanup
Decode Pool	Focuses on stable generation and bandwidth utilization; prioritizes protecting TPOT/ITL
Observability	Provides end-to-end tracing for TTFT, TPOT, KV transfer, queues, and failure rates

For small to medium-sized teams, the first version should not aim for complex cross-cluster KV storage. Prioritize a minimal closed loop within the same data center and high-speed network domain to validate tail-latency benefits, then gradually introduce more complex multi-level KV caching and heterogeneous resource pools.

Conclusion: P/D Disaggregation Is a Latency Management Architecture, Not a Single-Point Performance Toggle

The value of Prefill-Decode disaggregation is not in making the service more complex, but in acknowledging that the two phases of LLM inference have different resource profiles and different user experience metrics.

As applications enter long-context, multi-turn agent, multi-tenant, and high-concurrency stages, a single GPU pool can easily create trade-offs between TTFT, TPOT, throughput, and cost. P/D disaggregation offers a more controllable architecture: let prefill focus on processing context, let decode focus on stable generation, and let the KV cache become a first-class data plane connecting them.

But it also introduces new engineering costs. Before going live, you must answer three questions:

1. Is your tail-latency problem truly caused by prefill/decode interference?
2. Is your KV cache transfer link fast enough, observable enough, and recoverable enough?
3. Are your resource ratios based on real input/output length and SLO stress testing, not just educated guesses?

If the answers are yes, P/D disaggregation will be a key architectural step for long-context LLM services moving from “working” to “stable, scalable, and cost-controllable.”

---

Key References:

1. vLLM Documentation, Disaggregated Prefilling — https://docs.vllm.ai/en/latest/features/disagg_prefill/
2. Hao AI Lab, Disaggregated Inference: 18 Months Later (2025-11-03) — https://haoailab.com/blogs/distserve-retro/
3. Luchang Li et al., SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference (2026-03-05) — https://arxiv.org/abs/2603.04716
4. Ruihan Lin et al., Large-Scale LLM Inference with Heterogeneous Workloads (2026-02-03) — https://arxiv.org/abs/2602.02987
5. Hengrui Zhang et al., SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference (2025-10-09) — https://arxiv.org/abs/2510.08544
6. Yihua Cheng et al., LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference (2025-10-08) — https://arxiv.org/abs/2510.09665
7. Ruoyu Qin et al., Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter (2026-04-16) — https://arxiv.org/abs/2604.15039
8. Zongze Li et al., Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving (2026-03-09) — https://arxiv.org/abs/2603.13358