Is Prefill-Decode disaggregation always better than continuous batching?

No. It primarily addresses the problem of long prompt prefills interfering with decode. If requests are short, concurrency is low, or TTFT constraints are extremely tight, traditional aggregated serving may be simpler and more suitable.

What is the biggest engineering risk of disaggregation?

The core risks are KV Cache transfer, scheduling, and capacity ratio. An imbalanced ratio between prefill and decode instances, or high cross-node transfer latency, can negate the benefits of disaggregation.

Which metrics should I monitor most closely before going live?

At a minimum, you should simultaneously observe TTFT, TPOT/ITL, KV transfer latency, prefill queue depth, decode queue depth, SLO attainment rate, GPU utilization, and request length distribution.

Prefill-Decode Disaggregation in Practice: Governing TTFT and TPOT for Long-Context LLM Serving

Background: Why Long-Context Requests Degrade Interactive Experience

In large language model (LLM) online serving, a request typically consists of two phases: Prefill and Decode.

Prefill processes the input prompt and computes the KV Cache for the prompt tokens. This phase is compute-intensive; the longer the input, the more pronounced the latency.
Decode generates output tokens one by one, based on the already computed KV Cache. This phase is more sensitive to GPU memory bandwidth, KV Cache access patterns, and scheduling wait times.

In traditional deployments, prefill and decode usually run on the same set of GPUs. Techniques like Continuous Batching, PagedAttention, and Chunked Prefill can mitigate many issues. However, when a system handles a mix of long-context requests, a new long-prompt prefill can occupy the GPU compute window, squeezing ongoing streaming outputs. This manifests as:

Increased time-to-first-token (TTFT);
Jittery token generation intervals (TPOT / ITL);
A “stutter” effect in streaming conversations (pause, a few tokens, pause again);
Deceptively high GPU utilization but unstable end-user latency;
Over-provisioning to meet latency targets, raising the cost per token.

Prefill-Decode Disaggregation (PD Disaggregation) aims not simply to “make inference faster,” but to decouple these two phases with distinct computational characteristics into separate deployments. This allows prefill resources to focus on input processing and decode resources to focus on stable generation, enabling finer-grained governance of TTFT, TPOT, and overall SLOs.

Core Principle: Splitting Inference into Two Resource Pools

The basic flow of Prefill-Decode disaggregation is as follows:

A request enters the scheduling layer.
The scheduler selects a Prefill Worker.
The Prefill Worker processes the prompt and computes the KV Cache for the prompt tokens.
The Prefill Worker transfers the KV Cache (or its metadata) to a Decode Worker.
The Decode Worker uses this KV Cache to generate the output tokens in a streaming fashion.

Think of it this way: prefill is “building the index,” and decode is “querying the index and continuously writing the answer.” If these two tasks are mixed in a single GPU queue, a long prompt will interfere with ongoing generation. If decoupled, each phase can be scheduled, scaled, and optimized independently.

Why Prefill and Decode are Suitable for Disaggregation

Prefill and decode have different bottlenecks:

Phase	Primary Bottleneck	Impact on User Experience
Prefill	Batch matrix compute capacity	Determines TTFT (time user waits for first token)
Decode	Low jitter, sustained token generation	Determines TPOT / ITL (interaction fluency)

Therefore, a disaggregated architecture allows the system to configure different resource strategies for each phase. For example, the prefill pool can use parallelism schemes better suited for high-throughput prompt processing, while the decode pool uses schemes better suited for stable streaming output.

Key Cost: KV Cache Must Be Transferred Across Instances

Disaggregation is not free. The KV Cache computed by the Prefill Worker must be used by the Decode Worker, introducing new engineering costs:

KV Cache transfer latency;
Bandwidth pressure across GPUs, nodes, or racks;
KV block layout conversion;
Lifecycle coordination between prefill and decode instances;
KV cleanup upon request failure, cancellation, or timeout;
Scheduling consistency in multi-replica, multi-tenant scenarios.

Thus, the key to PD disaggregation is not simply “splitting the service into two processes,” but designing the KV Transfer, Router, Queue, SLO, and Backpressure mechanisms together.

Engineering Implementation: A Production-Ready Architecture

A production-grade PD disaggregation system typically comprises five layers.

1. Ingress Routing Layer

The ingress layer receives OpenAI-compatible API or internal RPC requests and selects the processing path based on request length, model, tenant, SLO, and historical load.

Typical strategies include:

Short prompt, short output: Route directly to standard aggregated serving.
Long prompt, short-to-medium output: Prefer PD disaggregation.
Long prompt, long output: Decide based on queue pressure.
TTFT-sensitive requests: Use disaggregation cautiously, as KV transfer may increase first-token latency.
TPOT / ITL-sensitive requests: Prefer disaggregation to prevent decode from being interrupted by prefill.

2. Prefill Worker Pool

The Prefill Worker is the KV Producer. It handles prompt encoding, KV Cache generation, and returning the first token or context metadata.

Production considerations:

Use a dedicated queue for long prompts to avoid starving short prompts.
Apply special scheduling for requests hitting the prefix cache to reduce redundant prefill.
Limit the maximum input length per request to prevent a single request from overwhelming the queue.
Record prefill queue time, prefill compute time, and cache hit ratio.
Release KV buffers promptly for timed-out requests.

3. KV Transfer Layer

KV Transfer is the core path of the disaggregated architecture. Different frameworks may use RDMA, UCX, NIXL, MPI, shared memory, or custom connectors.

Before going live, you must clarify:

Is the KV Cache transferred directly from GPU memory, or staged through CPU memory?
Can the transfer be overlapped with computation for other requests?
Do KV blocks need remapping under different parallelism strategies?
Does the Decode Worker pull the KV Cache, or does the Prefill Worker push it?
How to handle retries, degradation, or error reporting on transfer failure?

⚠️ A common mistake is to benchmark only prefill and decode performance while ignoring KV transfer. The offline benchmark may show clear benefits, but in production, gains can be eroded by network latency, serialization, block mapping, and cross-node scheduling.

4. Decode Worker Pool

The Decode Worker is the KV Consumer. Its goal is not to maximize prefill batch size, but to maintain stable token output.

Production strategies typically include:

Controlling decode batch size based on TPOT / ITL constraints.
Preventing decode queue buildup.
Setting reasonable budgets for long-output requests.
Reclaiming KV Cache promptly for cancelled requests.
Reserving decode capacity for high-priority interactive requests.
Monitoring p50, p90, and p99 TPOT, not just average tokens/s.

5. Global Scheduling and Capacity Ratio

The effectiveness of PD disaggregation heavily depends on the resource ratio between prefill and decode.

Too few prefill resources degrade TTFT; too few decode resources cause streaming stalls; over-provisioning both wastes GPUs. Instead of a fixed ratio, maintain several configurations based on business patterns:

Business Pattern	Prefill Resources	Decode Resources
Short Q&A	Low	Moderate
Long-document Q&A	High	Moderate
Code generation	Moderate	High
Agent / tool calling	Dynamic routing	Dynamic routing
Batch summarization	Can sacrifice interactivity	May not need disaggregation

When to Use PD Disaggregation

Suitable Scenarios

Long-context interaction: Long-document Q&A, contract analysis, codebase Q&A, multi-turn conversations with extensive history. Long prompts amplify prefill interference on decode; disaggregation helps maintain stable output.
High-concurrency streaming services: If users value real-time output and p99 ITL exceeds the SLO, PD disaggregation often provides more room for governance than continuing to stack Chunked Prefill.
Different parallelism strategies for different phases: For example, prefill may benefit from a specific tensor parallelism configuration, while decode benefits from another. Aggregated deployment forces a shared strategy; disaggregation allows independent optimization.
Multi-tenant SLO tiering: In systems mixing paid tiers, interactive, and batch scenarios, high-priority decode requests can get a more stable resource pool, reducing the chance of interference from long prompts.

Unsuitable Scenarios: Don’t Disaggregate for the Sake of Novelty

Short prompts, low concurrency: If requests are generally short, prefill interference is minimal. Splitting only adds deployment complexity and KV transfer cost.
TTFT is the sole core metric: Disaggregation often introduces scheduling and KV transfer overhead. If the business prioritizes the first token arriving instantly and is insensitive to subsequent output stability, aggregated serving may be better.
Insufficient network bandwidth: Cross-node KV Cache transfer requires a stable, low-latency link. If bandwidth between GPUs, nodes, or racks is limited, disaggregation benefits diminish rapidly.
Lack of fine-grained monitoring: Without metrics for TTFT, TPOT, KV transfer, queue depth, and request length distribution, you cannot determine if disaggregation truly improves SLOs. Blindly deploying it only increases debugging difficulty.

Common Misconceptions

Misconception 1: Focusing Only on Tokens/s, Ignoring SLOs

The core metric for PD disaggregation is not average throughput, but throughput under latency constraints. That is, how many requests can the system stably serve while meeting TTFT and TPOT targets?

A system with high average tokens/s but jittery p99 ITL is still unacceptable for interactive workloads.

Misconception 2: Routing All Requests Through the Disaggregated Architecture

Disaggregation should be one strategy, not the only entry point. Short requests, low-load requests, and TTFT-sensitive requests can continue using the aggregated path.

A more robust approach is dynamic routing based on request length, expected output length, tenant tier, SLO type, and current queue pressure.

Misconception 3: Ignoring the KV Cache Lifecycle

The KV Cache is not a simple intermediate variable. It can be very large, and its lifecycle is tightly coupled with the request state. Cancellation, timeout, decode failure, client disconnection, and worker restarts all involve KV cleanup.

Without a robust reclamation strategy, PD disaggregation can introduce subtle memory leaks and cache pollution.

Misconception 4: Unrealistic Benchmark Data Distribution

The benefits of PD disaggregation are most evident under long-context, high-concurrency, strict TPOT scenarios. Benchmarking with only uniform prompt lengths or fixed output lengths can easily lead to misleading conclusions.

Benchmark data should cover at least: short prompts, long prompts, short outputs, long outputs, streaming interruptions, timeouts, traffic bursts, and both prefix cache hits and misses.

Go-Live Checklist

Architecture Check

Are the prefill pool and decode pool clearly separated?
Is an aggregated path available as a fallback?
Can the system choose to disaggregate or not based on request characteristics?
Can the P/D resource ratio be adjusted without a full system restart?
Are worker registration, removal, health checks, and canary deployment strategies defined?

KV Transfer Check

Is KV transfer latency being recorded?
Are transfer failure retries or fast-fail mechanisms supported?
Is the mapping between KV block layout and parallelism strategy clear?
Has latency been verified across nodes, NUMA domains, and racks?
Are KV Cache cleanup and leak detection mechanisms in place?

SLO Check

Are separate TTFT and TPOT / ITL targets defined?
Are metrics observed at p90 / p99, not just averages?
Is the SLO attainment rate being tracked?
Can metrics be bucketed by model, tenant, and request length?
Are overload rejection, degradation, and queuing strategies defined?

Benchmark Check

Is the request length distribution realistic?
Does it cover long prompts under high concurrency?
Does it cover streaming output scenarios?
Are client cancellation and timeout scenarios tested?
Are the three paths (aggregated, chunked prefill, PD disaggregation) compared?

A Simple Routing Strategy Example

The following example illustrates the strategy structure, not a complete production implementation:

def choose_serving_path(
    input_tokens: int,
    expected_output_tokens: int,
    ttft_sensitive: bool,
    current_decode_itl_p99_ms: float,
    prefill_queue_depth: int,
) -> str:
    if input_tokens < 2048 and current_decode_itl_p99_ms < 120:
        return "aggregated"
    if ttft_sensitive and input_tokens < 4096:
        return "aggregated"
    if input_tokens >= 8192 and expected_output_tokens <= 1024:
        return "disaggregated_pd"
    if current_decode_itl_p99_ms > 180 and prefill_queue_depth > 0:
        return "disaggregated_pd"
    return "chunked_prefill"

The key point of such a strategy is not the specific thresholds, but transforming routing from a “fixed architecture choice” into an “online decision based on SLOs and load.”

FAQ

Q: Are Prefill-Decode disaggregation and Chunked Prefill alternatives?

No. Chunked Prefill splits a long prefill into smaller chunks within the same serving instance to reduce blocking of decode. PD disaggregation places prefill and decode into different resource pools. They are complementary and can be chosen based on workload.

Q: Will TTFT always worsen after disaggregation?

Not necessarily, but it requires careful evaluation. Disaggregation adds scheduling and KV transfer paths. If these overheads cannot be overlapped with computation, TTFT may increase. However, in long-prompt, high-concurrency scenarios, reducing queue interference can also improve tail TTFT.

Q: How to determine if the resource ratio is reasonable?

Watch two queues: the prefill queue and the decode queue. If the prefill queue is persistently backlogged, prefill resources are insufficient. If decode ITL jitters and the decode queue is backlogged, decode resources are insufficient. The resource ratio should be dynamically adjusted based on the real request length distribution, not set once.

Conclusion

Prefill-Decode disaggregation is an architectural optimization that truly demonstrates its value only when long-context LLM serving reaches scale. It addresses not single-request inference speed, but the resource interference problem arising from the combination of long prompts, streaming decode, high concurrency, and strict SLOs.

Avoid two extremes during implementation: treating it as a universal accelerator for all requests, or dismissing it solely due to deployment complexity while ignoring the real damage long-context requests inflict on decode stability.

A more sensible path is: first, use real traffic to identify bottlenecks in TTFT, TPOT, request length, and queues. Then, introduce PD disaggregation as a routable, degradable, and observable serving path. Only when KV Transfer, Router, queues, and SLO monitoring are closed together does PD disaggregation transform from a paper architecture concept into a production tool for governing cost and user experience.