Continuous Batching in Practice: Why Traditional Fixed Batching Falls Short for LLM Serving

Background: Why Traditional Fixed Batching Doesn’t Work for Online LLM Serving

In traditional deep learning inference, the intuition behind batching is simple: group multiple requests into a batch, feed them into the GPU at once, and increase parallelism. However, text generation with large models isn’t a single forward pass. A request typically involves two phases:

1. Prefill Phase: Process the full prompt and compute the initial KV Cache.
2. Decode Phase: Generate one or a few tokens per step, continuously reading and appending to the KV Cache.

The problem is that different user requests vary greatly in prompt length, output length, and arrival time. With traditional fixed batching, systems usually encounter three core issues:

• Short requests are held back by long ones: Requests in the same batch need to execute roughly synchronously; even if a short request finishes, it may have to wait until the entire batch completes before its slot is freed.
• New requests wait for the batch to finish: While a batch is running, newly arrived requests cannot immediately enter the GPU and must wait in a queue.
• Significant padding and KV Cache waste: To align sequences of different lengths, the system incurs substantial useless computation or memory consumption.

The ORCA paper describes this problem directly: generative Transformer inference is iterative by nature. Traditional scheduling cannot change the set of requests while a batch is running, so completed requests cannot be returned promptly, and new requests must wait for the current batch to fully complete. ORCA’s core improvement is iteration-level scheduling, which reorganizes the batch at each generation iteration.

This is precisely the starting point for Continuous Batching.

The Core Idea of Continuous Batching

Continuous Batching, also often called in-flight batching or iteration-level batching, doesn’t simply increase the batch size. It fundamentally changes the scheduling granularity.

Traditional batching operates at the “request level”:

batch = [request_a, request_b, request_c]
run until all requests finish
return results
start next batch

Continuous Batching operates at the “generation iteration level”:

while server_running:
    remove_finished_requests()
    add_new_requests_if_capacity_available()
    schedule_prefill_and_decode_tokens()
    run_one_or_more_model_steps()
    stream_completed_tokens()

This means the system can dynamically adjust the set of active requests at each step of the token generation loop: finished requests exit immediately, and new requests can enter as long as there is capacity. TensorRT-LLM’s Batch Manager documentation also describes in-flight batching as a mechanism to reduce queue wait times, eliminate padding requests, and improve GPU utilization, allowing new requests to be included and completed requests to be returned at each iteration of the token generation loop.

From an engineering perspective, Continuous Batching is not a single-point optimization but a combination of capabilities:

Capability	Description
Request Pool Management	Maintains states like waiting, running, finished, and aborted
Iteration-level Scheduling	Decides which requests execute prefill and which execute decode at each round
Token Budget Control	Limits the total number of tokens processed per round to prevent long prompts from monopolizing resources
KV Cache Allocation & Reclamation	Dynamically manages KV Cache as requests enter, grow, complete, or are cancelled
Streaming Output	Pushes generated tokens to the client as soon as possible during the decode phase, rather than waiting for the entire batch to finish

Why PagedAttention Often Accompanies Continuous Batching

Continuous Batching solves the problem of “how requests enter and leave a batch,” but it amplifies another issue: KV Cache grows dynamically and has highly irregular lifecycles.

Each request maintains its own KV Cache. The longer the request, the larger the KV Cache. When a request completes, is cancelled, or times out, its KV Cache must be freed. If managed as contiguous blocks of memory, the system is prone to fragmentation. With significant fragmentation, available memory may appear sufficient but cannot accommodate new requests, ultimately limiting batch size and throughput.

PagedAttention borrows the idea of virtual memory paging from operating systems, managing KV Cache in fixed-size blocks. The PagedAttention paper points out that the KV Cache in LLM serving is large and dynamically grows and shrinks; inefficient management wastes significant memory due to fragmentation and redundant copying, thereby limiting batch size. PagedAttention achieves near-zero waste through block-level management and supports sharing KV Cache within and across requests.

The relationship between the two can be understood as follows:

• Continuous Batching makes the set of requests dynamically change during runtime.
• PagedAttention makes the dynamically changing KV Cache easier to allocate, reuse, and release.
• Combining both is essential for stably improving throughput in real-world services with high concurrency and a mix of long and short requests.

This is why vLLM simultaneously emphasizes capabilities like PagedAttention, continuous batching, chunked prefill, and prefix caching. Judging any single optimization in isolation can be misleading; in production, they must be evaluated together within the same scheduling system.

The Lifecycle of a Request in a Continuous Batching System

1. Admission: Entering the Waiting Queue

When a request arrives, it doesn’t execute immediately. It first enters a waiting queue. The scheduler needs to check:

• Whether there are available KV Cache blocks on the GPU.
• Whether the current number of active requests has reached the limit.
• Whether the current token budget allows for the new prompt tokens.
• Whether the request has priority, timeout, tenant, or SLA constraints.

Controlling capacity solely with max_batch_size is problematic. For LLM services, it’s better to control both the number of requests and the number of tokens. A single 32K prompt request places far more stress on the system than dozens of short prompt requests.

2. Prefill: Computing the Context

The prefill phase is typically compute-heavy. Long prompts consume significant compute and increase TTFT (Time To First Token). If continuous batching stuffs a large number of long prompts into the GPU at once, decode requests can be squeezed, causing users currently receiving streaming output to experience “stuttering.”

Therefore, production environments often combine Continuous Batching with chunked prefill: long prompts are not processed all at once but are split into multiple chunks and interleaved with decode requests. This reduces the blocking of decode by prefill.

3. Decode: Generating Tokens One by One

The decode phase is usually more memory-bound, as each step requires reading the existing KV Cache. One of the main benefits of Continuous Batching is maintaining high concurrency during the decode phase: when some requests finish, new ones can fill in, preventing the GPU from idling while waiting for a few long-output requests at the tail of the batch.

However, cramming more requests into decode isn’t always better. With too many active sequences, the pressure on KV Cache reads increases, and ITL (Inter-Token Latency) and tail latency can degrade.

4. Completion / Abort: Releasing Resources

When a request completes, the client disconnects, times out, or is cancelled, the system must promptly release:

• KV Cache blocks.
• The running slot in the scheduler.
• Streaming connection resources.
• Trace, metrics, and logging context.

This step is often underestimated. If cancelled requests don’t release their KV Cache in time, GPU memory utilization can spike abnormally, affecting the admission of subsequent requests.

Common Pitfalls in Production Deployment

Pitfall 1: Focusing Only on Throughput, Ignoring TTFT and ITL

Continuous Batching can easily boost total throughput, but throughput isn’t the only metric. For online services, you should at least break down the metrics:

Metric	Meaning
TTFT	How long until the user sees the first token
ITL	Whether subsequent token intervals are stable
TPOT	Average time per output token
P95/P99 latency	Whether tail requests are slowed down
Queue waiting time	Whether latency comes from queuing or model execution

If you only look at tokens/s, the system might look great in benchmarks, but real users will experience slow first tokens or jittery streaming output.

Pitfall 2: Treating max_batch_size as the Only Knob

Traditional inference often tunes batch size, but LLM serving requires tuning a set of constraints:

scheduler:
  max_num_seqs: 128
  max_num_batched_tokens: 8192
  max_model_len: 32768
  prefill_chunk_size: 2048
  kv_cache_utilization_limit: 0.85
  waiting_timeout_ms: 2000

These parameters interact with each other:

• max_num_seqs too large → high decode concurrency, but more KV Cache pressure.
• max_num_batched_tokens too large → high prefill throughput, but may suppress decode.
• max_model_len too large → high single-request limit, but more conservative admission.
• kv_cache_utilization_limit too aggressive → prone to OOM or frequent rejections under tail load.

Pitfall 3: Directly Porting Benchmark Results to Production

Benchmark scripts often use fixed input lengths, fixed output lengths, and fixed arrival rates. But in real business, request distributions are typically long-tailed:

• Some users ask just one sentence.
• Some Agents inject tens of KB of tool context.
• Some tasks output a dozen tokens.
• Some tasks output thousands of tokens.
• Some clients disconnect mid-stream.

Therefore, production stress tests shouldn’t only run “average lengths.” You should at least construct scenarios for short prompts, long prompts, short outputs, long outputs, mixed arrivals, traffic spikes, and request cancellations.

Pitfall 4: Ignoring Multi-Tenant Fairness

Continuous Batching makes the system more aggressive in filling the GPU, but without fair scheduling, large tenants or long-prompt tasks can squeeze out smaller ones. Common practices include:

• Setting concurrency limits per tenant.
• Setting token budgets per request type.
• Separating queues for interactive requests and offline tasks.
• Applying stricter admission policies for long-context requests.
• Using deadline-aware or SLA-aware scheduling strategies.

Pre-Production Checklist

Scheduling Parameters

• Are both the active request limit and the batched token limit set?
• Is the maximum input length and maximum output length per request limited?
• Are prefill token budget and decode token budget differentiated?
• Is chunked prefill enabled or evaluated?
• Have resource release tests been performed for request cancellations, timeouts, and client disconnections?

Memory and KV Cache

• Are KV Cache block utilization, free blocks, and reclamation speed monitored?
• Are the reasons for admission failures monitored: insufficient memory, insufficient token budget, full concurrency, timeout?
• Have stress tests been run with mixed long-context traffic?
• Have the combined effects of features like prefix caching, LoRA, and speculative decoding with the scheduler been validated?

Latency Metrics

• Are TTFT, ITL, TPOT, queue time, and model execution time broken down?
• Are P50, P95, and P99 observed, rather than just averages?
• Is the impact of long prompts on short requests separately observed?
• Is the jitter of streaming tokens under high load observed?

Fault Tolerance and Degradation

• When GPU memory is near its threshold, are new requests rejected instead of causing an OOM?
• Is rate limiting supported per tenant, model, or request type?
• Is it possible to quickly reduce max_num_seqs or token budget?
• Is a trace ID retained during anomalies to facilitate locating the specific request?

A More Prudent Tuning Sequence

Production tuning shouldn’t aim for maximum throughput from the start. It’s recommended to proceed in the following order:

Step 1: Establish a Baseline

Start with conservative settings to get the service running:

max_num_seqs: 32
max_num_batched_tokens: 4096
max_model_len: 8192
kv_cache_utilization_limit: 0.75

Record TTFT, ITL, throughput, and memory utilization under different scenarios.

Step 2: Increase Decode Concurrency

Gradually increase max_num_seqs and observe whether ITL degrades. If throughput improves significantly and ITL remains stable, continue increasing. If ITL becomes more volatile, the decode phase is likely approaching a bottleneck.

Step 3: Control Prefill Impact

Gradually adjust max_num_batched_tokens and chunked prefill parameters. The goal isn’t to maximize prefill throughput in one go, but to prevent prefill from blocking decode.

Step 4: Introduce Real-World Distributions

Construct stress test data using samples from real logs, including at least:

• Input length distribution.
• Output length distribution.
• Arrival rate distribution.
• Request cancellation ratio.
• Tenant distribution.
• Tool call or Agent context distribution.

Step 5: Set Protective Guardrails

Define clear protective guardrails for the online service:

slo:
  ttft_p95_ms: 1500
  itl_p95_ms: 120
  queue_wait_p95_ms: 500
  kv_cache_utilization_max: 0.88
  admission_reject_rate_max: 0.01

When metrics trigger these guardrails, prioritize degrading admission or long-context requests rather than continuing to expand the batch.

Applicable Scenarios and Boundaries

Suitable Scenarios

• Online chat, code completion, Agent services with continuous request arrivals.
• Mixed traffic with significant variance in input and output lengths.
• Streaming output services.
• Services with low GPU utilization and obvious queue wait times.
• Services needing a dynamic balance between throughput and latency.

Scenarios Requiring Caution

• Very low-latency, low-concurrency small model services — scheduling overhead may negate benefits.
• Services with a very high proportion of ultra-long contexts, without chunked prefill or prefix caching.
• Multi-tenant strong isolation scenarios lacking fair scheduling.
• Services already under severe memory pressure with immature KV Cache management.
• Services using complex combined optimizations (Multi-LoRA, speculative decoding, guided decoding) without independent stress testing.

Conclusion

The value of Continuous Batching isn’t about “increasing the batch size.” It’s about transforming LLM generation services from request-level fixed batching to iteration-level dynamic scheduling. It allows completed requests to exit promptly, new requests to enter quickly, and, combined with KV Cache management, reduces memory waste.

However, it also exposes scheduling, memory, SLO, and fairness issues more clearly. A truly production-ready Continuous Batching system needs to simultaneously manage prefill, decode, KV Cache, token budgets, request cancellations, and multi-tenant policies. For long-context and Agent services, it’s recommended to design it alongside PagedAttention, chunked prefill, prefix caching, and SLA-aware admission, rather than treating it as an isolated optimization.

Key References

• vLLM Documentation: https://docs.vllm.ai/en/latest/
• TensorRT-LLM Overview: https://nvidia.github.io/TensorRT-LLM/overview.html
• TensorRT-LLM GPT Attention / In-flight Batching: https://nvidia.github.io/TensorRT-LLM/advanced/gpt-attention.html
• TensorRT-LLM Batch Manager: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/batch_manager.md
• Hugging Face TGI: https://huggingface.co/docs/text-generation-inference/en/index
• ORCA (OSDI ‘22): https://www.usenix.org/conference/osdi22/presentation/yu
• PagedAttention (arXiv): https://arxiv.org/abs/2309.06180
• Memory-aware Dynamic Batching (arXiv): https://arxiv.org/abs/2503.05248
• Ragged Paged Attention (arXiv): https://arxiv.org/abs/2604.15464