Is Chunked Prefill the same as regular Continuous Batching?

No. Continuous Batching handles dynamic request entry/exit at the batch level, while Chunked Prefill further splits long input prefills into smaller chunks, allowing decode requests to interleave.

Is a smaller chunk size always better?

Not necessarily. Smaller chunks usually improve ITL but can increase prefill iterations and worsen TTFT. Larger chunks favor prefill throughput but may block decode.

When should I consider P/D disaggregation instead of tuning Chunked Prefill?

When long-context requests persistently impact decode tail latency, and chunk size within a single instance cannot balance TTFT and ITL, evaluate prefill/decode disaggregation.

Chunked Prefill in Production: Latency Governance for Long-Context Inference

Background: Why Long-Context Requests Slow Down Online Inference

Online inference for large models typically involves two phases: prefill and decode. The prefill phase processes all input tokens at once, generating the KV Cache needed for subsequent decoding. The decode phase generates output tokens one by one. These phases utilize the GPU differently—prefill is more compute-bound, easily saturating matrix compute units, while decode is more memory-bound, processing only the new token each step but reading an ever-growing KV Cache.

In short-prompt scenarios, this difference is not pronounced. However, with long contexts, Agent toolchains, RAG concatenation, or poorly compressed multi-turn dialogues, prefill can become a major source of disruption for online services. When a request with tens of thousands of tokens enters the queue, if the scheduler executes it as a complete prefill in one go, ongoing decode requests may be forced to wait. Users experience not just a drop in average throughput, but inter-token latency (ITL) jitter, stuttering streaming output, and increased tail latency.

The problem is even more evident with traditional static batching: a batch may be held up by a long request even after some requests have finished, and new requests must wait until the current batch completes. Orca, presented at OSDI 2022, introduced iteration-level scheduling, reducing the scheduling granularity from the request level to the iteration level. This allows completed requests to return promptly and new requests to join the batch in the next iteration. This is a foundational idea behind today’s Continuous Batching.

But Continuous Batching alone is not enough. It solves the problem of “how requests dynamically enter and leave a batch,” but it doesn’t fully address whether a single, very long prefill will consume too much of the token budget in a given iteration. This is the problem Chunked Prefill aims to solve.

Core Principle: Chunking Long Prefills to Prevent Decode Starvation

Continuous Batching Fills Batch Gaps

The core of Continuous Batching is: don’t wait for an entire batch to finish before swapping in the next one. Instead, re-check the queue after every generation iteration. When a request completes, release its slot; when a new request arrives in the queue, add it as soon as possible. The Hugging Face TGI documentation also highlights Continuous Batching and streaming as core capabilities of a production inference engine, emphasizing the dynamic composition of in-flight requests and streaming tokens back via SSE.

It primarily solves two problems:

Short-output requests are no longer held up by long-output requests.
The GPU batch remains as full as possible, reducing the gaps (idle slots) seen in static batching.

However, Continuous Batching still faces a sharp problem with long-context inputs: if a request’s prefill is very large, it consumes too much of the token budget in a single iteration. While decode requests can theoretically be inserted, they may still be slowed down by the prefill in practice.

Chunked Prefill Solves Long-Input Blocking

Chunked Prefill works by splitting a large prefill into multiple smaller chunks. The vLLM documentation clearly states that chunked prefill divides large prefills into smaller chunks and batches them together with decode requests, achieving a better balance between compute-bound prefill and memory-bound decode.

In vLLM V1, chunked prefill is enabled by default when possible; the scheduling policy prioritizes pending decode requests, then allocates the remaining max_num_batched_tokens budget to prefill. If a prefill doesn’t fit in the budget, the system automatically chunks it.

Think of it as a simple scheduling strategy:

while serving:
    batch = []
    # 1. First, protect streaming decode requests
    batch.extend(pending_decode_requests)
    # 2. Then, fill the remaining token budget with prefill work
    remaining = max_num_batched_tokens - tokens(batch)
    while remaining > 0 and has_pending_prefill():
        chunk = next_prefill_chunk(max_tokens=remaining)
        batch.append(chunk)
        remaining -= tokens(chunk)
    run_one_iteration(batch)

The key point of this strategy isn’t to “make prefill faster,” but to prevent prefill from blocking the scheduling loop as a single, complete long request. Decode requests get more frequent execution opportunities, leading to more stable streaming output.

Why Chunk Size is the Core Knob

The primary tuning parameter for Chunked Prefill is usually the token budget or chunk size. In vLLM, a key parameter is max_num_batched_tokens. The documentation provides clear guidance:

Strategy	Advantage	Cost
Smaller Chunk	Improves ITL, less interference with decode	Increases prefill iterations, may worsen TTFT
Larger Chunk	Reduces TTFT, processes more prefill per batch	May block decode, impacting tail latency

NVIDIA’s TensorRT-LLM technical blog on chunked prefill also notes that a larger chunk size reduces the number of iterations needed to process a prefill sequence, thus lowering TTFT. However, it also increases the completion time for ongoing decode phases, affecting query completion time and output TPS. Therefore, chunk size is neither strictly better when larger nor smaller; it’s a trade-off determined by business objectives.

Engineering Implementation: From Metrics and Parameters to Deployment Strategy

First, Decompose Service Objectives

Before going live, don’t just look at tokens/s. Break down your metrics into at least four categories:

TTFT (Time to First Token): The time a user waits for the first token, primarily affected by prefill, queuing, and scheduling policy.
ITL (Inter-Token Latency): The interval between consecutive tokens during streaming output, directly impacting perceived “smoothness” or “stuttering.”
End-to-end Latency: The total time to complete a full response, influenced by output length, decode speed, and queuing.
Goodput: The throughput of requests that successfully meet their SLO (Service Level Objective), not the maximum throughput under unconstrained load testing.

Many inference optimizations perform well under low-concurrency load tests but reveal problems under mixed workloads. For example, when RAG requests, regular chat requests, and long Agent trace requests are mixed in the same instance, the average tokens/s might look fine, but the P95/P99 ITL could be unacceptable.

Then, Determine the Tuning Order

It is recommended to proceed in the following order:

1. Fix the model, quantization method, and concurrency load test script.
2. First, enable Continuous Batching / default scheduling capabilities.
3. Then, tune max_num_batched_tokens or the equivalent token budget.
4. Observe TTFT, ITL, P95/P99 latency, and OOM/preemption events separately.
5. Finally, layer on Prefix Caching, KV Offloading, and P/D Disaggregation.

In vLLM, you can specify the token budget like this:

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    max_num_batched_tokens=8192,
)

This value is not a universal answer. For smaller models, larger VRAM, and short-output scenarios, you can try a larger token budget. For long-context, multi-tenant, or strongly interactive streaming scenarios, pay more attention to ITL and tail latency, which typically requires a more conservative chunk configuration.

Monitor Preemption and Queue Backlog

Chunked Prefill is not an infinite expansion of VRAM. Long-context requests still consume KV Cache, and uncertain output lengths can still cause peak VRAM usage to exceed estimates. The vLLM documentation in its optimization section also reminds users to monitor preemption requests via Prometheus metrics. During deployment, you should also observe:

Whether the prefill queue length is growing continuously.
Whether the decode queue is intermittently blocked by long prefills.
Whether preemption is increasing.
Whether KV Cache usage is consistently near its limit.
Whether P99 ITL is excessively higher than P50 ITL.

If preemption is frequent, the problem likely isn’t just about tuning chunk size anymore. It may require redesigning concurrency limits, context length handling, KV Cache strategies, or instance isolation.

Applicable Scenarios

Scenarios Suitable for Chunked Prefill

Long contexts requiring a streaming experience: e.g., RAG, code repository Q&A, document analysis, Agent task replay. Users care not only about the final completion time but also about the continuity of the output.
Significant mix of request lengths: Short chats and long prompts share the same set of GPUs. Without Chunked Prefill, long requests easily create tail latency.
P/D disaggregation is not yet necessary within a single instance: When the business scale isn’t complex enough to warrant separate prefill and decode clusters, Chunked Prefill is a lighter-weight governance tool.

Scenarios Where Chunked Prefill Alone is Insufficient

Very high proportion of extremely long contexts: If most requests are hundreds of thousands of tokens long, it’s very difficult for a single-instance scheduler to balance TTFT and ITL. Evaluate context compression, retrieval pruning, KV Cache tiering, and P/D disaggregation.
Strict low-latency conversational services: If the business requires very stable per-token latency and P99 ITL is a hard requirement, simply tuning chunk size may not be enough. Request prioritization, instance isolation, or a decode-priority queue may be needed.
Offline batch processing: If streaming output is not needed and TTFT is not a concern, aggressively pursuing Chunked Prefill might add unnecessary scheduling complexity. Offline tasks focus more on overall throughput and resource utilization.

Common Misconceptions

Misconception 1: Enabling Continuous Batching is a Silver Bullet

Continuous Batching only solves the problem of dynamic batch entry/exit. Without chunking, a long prefill can still block decode in a single iteration. For long-context services, Continuous Batching and Chunked Prefill should be considered together.

Misconception 2: Focusing Only on Throughput, Ignoring ITL

What users perceive most strongly is not tokens/s, but whether the streaming output pauses. A system might have excellent tokens/s, but if the output frequently stutters, users will still perceive the service as slow.

Misconception 3: Treating Chunk Size as a Static Optimal Value

Chunk size is related to model size, GPU type, concurrency, input length distribution, and output length distribution. NVIDIA’s materials also mention the value of dynamic chunk sizing: manually finding the balance point is not easy, and dynamic strategies can suggest more appropriate configurations based on information like GPU utilization.

Misconception 4: Ignoring Context Governance at the Application Entry Point

The inference layer can only mitigate problems, not replace upstream governance. Excessive RAG concatenation, unbounded tool trace growth, and repeated injection of system prompts all shift the pressure to prefill. Chunked Prefill is not an excuse to tolerate unbounded context lengths.

Go-Live Checklist

Load Test Data Preparation

Before going live, construct at least three types of workloads:

Short input, short output: Simulates regular chat.
Long input, short output: Simulates RAG, document Q&A, code retrieval.
Long input, long output: Simulates report generation, Agent summarization, complex reasoning.

Don’t just test with a single input length. The most dangerous scenario in production is a mixed distribution, especially the interference of a few long prompts on a large number of short requests.

Metrics Dashboard

It is recommended to place these metrics on a single dashboard:

QPS / active requests / waiting requests
TTFT P50/P95/P99
ITL P50/P95/P99
Output tokens/s and request/s
GPU SM utilization, HBM bandwidth, KV Cache usage
Preemption count, OOM count, request cancellation count
Latency distribution for different input length buckets

Rollback Conditions

It is recommended to define rollback conditions in advance, rather than discussing them during an incident:

P99 ITL exceeds the target threshold for 10 consecutive minutes.
Preemption increases significantly, accompanied by rising TTFT.
Long-context requests cause the P95 latency of short requests to double.
GPU utilization increases but goodput decreases.
Error rate, cancellation rate, or timeout rate exceeds the baseline.

Relationship with Other Technologies

Chunked Prefill vs. PagedAttention: These solve different problems. PagedAttention focuses more on KV Cache memory management and block-level scheduling to reduce fragmentation and increase concurrency. Chunked Prefill is more about the scheduling layer, breaking long prefills into smaller execution units to avoid blocking decode. In production, they are typically used together.

When to Consider P/D Disaggregation: When a small number of long-context requests consistently impact the experience of short requests, consider routing isolation based on input length, user tier, or task type. Chunked Prefill can mitigate interference, but not all mixed workloads are suitable for sharing the same pool.