What is the difference between Continuous Batching and regular dynamic batching?

Regular dynamic batching collects requests into a batch and executes the entire batch to completion. Continuous Batching reorganizes the batch at each decoding iteration, allowing completed requests to exit and new requests to enter promptly.

Does Continuous Batching always reduce per-request latency?

Not necessarily. It primarily improves GPU utilization and overall throughput. Per-request latency is still affected by queuing, prefill length, output length, batch size limits, and scheduling policies.

What should be monitored most when deploying Continuous Batching?

At minimum, monitor TTFT, TPOT, request queue time, number of running sequences, GPU utilization, KV Cache usage ratio, OOM count, and rejected request count.

Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

Background: The Bottleneck in LLM Serving Isn’t Just Slow Models

Large model online inference may seem like a simple interface—users send a request, the server returns tokens. But in production, the real cost comes from the interplay of GPU compute cycles, KV Cache memory, request queuing time, and long-tail output lengths.

Traditional inference services often follow the batching approach of conventional deep learning: collect a batch of requests and execute them together. This works well for image classification, embeddings, or fixed-length models because each sample has a relatively stable compute shape. However, text generation is autoregressive decoding—each request has different input and output lengths. Some requests finish in a few steps, while others generate hundreds of tokens.

If the server still operates on a “start together, wait together” basis, GPU efficiency is dragged down by two types of waste:

Short requests finish early, but their batch slots cannot be immediately reused by new requests.
To align different sequence lengths, the system may process significant padding or wait for the slowest request.

Continuous Batching addresses this by moving the scheduling granularity from the “request level” to the “decoding iteration level,” allowing the server to reorganize the set of running requests at each generation step.

Core Principle: From Request-Level Scheduling to Iteration-Level Scheduling

Continuous Batching is also often called iteration-level scheduling or in-flight batching. The ORCA paper articulates this clearly: each decoding step of a generative Transformer is an iteration. The scheduler does not need to pin a request to the same batch from start to finish; instead, it can decide at each iteration boundary which requests form the current batch.

A simplified execution flow is as follows:

Requests enter a queue, first undergo prefill to generate the initial KV Cache.
The scheduler selects a set of runnable requests into a decode batch.
The model executes one decode forward pass, typically generating the next token for each sequence.
Completed requests exit, releasing their KV Cache and batch slots.
New or waiting requests fill the slots for the next decode round.
Repeat steps 3–5 until requests finish or limits are reached.

The core of this mechanism is not “increase batch_size,” but allowing the batch to change dynamically during execution. The system no longer waits for an entire batch to finish; instead, it schedules at each token generation cycle. This keeps the GPU less idle and prevents requests from waiting for a fixed batch lifecycle before they can start execution.

Hugging Face Text Generation Inference explicitly lists continuous batching as a core capability for boosting total throughput; TensorRT-LLM uses the term in-flight batching, emphasizing dynamic management of context and generation phases to improve GPU utilization and reduce queuing.

Engineering Implementation: It’s Not Just Flipping a Switch

In inference services like vLLM, TGI, and TensorRT-LLM, Continuous Batching is often built-in, but production deployment still requires configuration around several control surfaces.

1. Distinguish Prefill and Decode

The prefill phase processes the full prompt, with compute typically proportional to input length. The decode phase generates a few tokens per round but executes repeatedly. Continuous Batching most directly optimizes decode phase slot utilization, but heavy prefill can still crush TTFT.

In production, don’t just look at tokens/s; break it down:

Metric	Composition
TTFT	queue waiting + prefill scheduling + first token generation
TPOT	decode scheduling + model forward + sampling + streaming overhead

If long prompts are common, simply improving decode batch efficiency won’t solve slow first tokens. You may also need:

Prefix cache to accelerate prefill.
Separate prefill/decode deployment.
Request throttling or prompt length tiering.

2. Control Maximum Concurrent Sequences, Not Blindly Increase Batch Size

Continuous Batching allows more requests to reside on the GPU simultaneously, but each request requires KV Cache. Higher concurrency increases memory pressure. Production parameters should be stress-tested around the following metrics:

serving_tuning:
  max_running_requests: 64
  max_total_tokens: 32768
  max_input_tokens: 8192
  max_output_tokens: 1024
  admission_control: true
  overload_policy: "queue_or_reject"

This configuration is not a fixed format for any framework but represents a capacity planning mindset: limit the number of running requests, total token budget, per-request max input/output lengths, and define an overload policy.

3. Coordinate with Streaming Output

Continuous Batching is often used with streaming. Streaming reduces perceived user wait time but introduces network flush, connection management, and client consumption speed differences. The server must prevent slow clients from holding resources. Common practices include:

Output buffer limits.
Connection timeouts.
Cancellation request reclamation.
KV Cache release upon client disconnection.

Applicable Scenarios: High Concurrency, Long Outputs, Low GPU Utilization

Continuous Batching is suitable for:

Multi-user shared model services with significant QPS fluctuations.
Large output length variance, where static batches are easily held up by long requests.
Unstable GPU utilization but high request queuing time.
Online chat, code completion, agent backends, batch content generation, and other continuous token generation businesses.
Inference frameworks already supporting dynamic scheduling, such as vLLM, TGI, and TensorRT-LLM.

It should not be understood as a “universal solution for all latency problems.” If the bottleneck is network transfer, upstream serial calls, slow model loading, saturated CPU tokenizer, or low request volume leaving the GPU idle, the benefits of Continuous Batching diminish.

Common Misconceptions

Misconception 1: Throughput Improvement Equals User Experience Improvement

Throughput improvement usually means more tokens per GPU, but user experience depends more on TTFT, TPOT, and tail latency. If concurrency is pushed too high to maximize throughput, short requests may queue longer, and P95/P99 latency can worsen.

Misconception 2: Bigger Batch Is Always Better

Larger batches can increase GPU utilization but also increase memory usage, scheduling complexity, and per-iteration time. The optimal point usually comes from stress testing, not experience. You need to test combinations of model size, prompt length, output length, GPU model, and business SLOs.

Misconception 3: Ignoring KV Cache Memory Growth

Continuous Batching allows more requests to run concurrently, and KV Cache usage grows with input length and generated tokens. Without admission control, systems are prone to OOM during traffic spikes, causing overall service instability.

Misconception 4: Only Looking at Averages

LLM Serving problems often hide in the long tail. Average tokens/s may look good, but that doesn’t mean user requests are stable. Before going live, you must check P95/P99 TTFT, TPOT, queue time, and cancellation rate.

Deployment Checklist

Capacity and Stress Testing

Use real prompt length distributions, not just short prompts.
Cover short output, long output, and mixed output requests.
Test at low, medium, and near-saturation QPS levels.
Record maximum sustainable tokens/s, not just peak values.
Test streaming and non-streaming outputs separately.

Key Monitoring Metrics

Metric	Meaning
`request_queue_time_seconds`	Request queuing wait time
`llm_ttft_seconds`	Time to first token
`llm_tpot_seconds`	Average time per output token
`running_requests`	Number of currently running requests
`waiting_requests`	Number of waiting requests
`gpu_utilization`	GPU utilization
`kv_cache_usage_ratio`	KV Cache usage ratio
`prefill_tokens_per_second`	Prefill phase throughput
`decode_tokens_per_second`	Decode phase throughput
`request_abort_total`	Total aborted requests
`request_oom_total`	Number of OOM events

KV Cache usage ratio and waiting_requests are particularly important. The former indicates whether the service is near memory risk; the latter reflects whether the scheduler is entering a congested state.

Overload Protection

Limit maximum input tokens.
Limit maximum output tokens.
Set different priorities or queues for different businesses.
Route extremely long requests separately.
Reject requests when thresholds are exceeded, rather than slowing all requests together.
Cancel generation and release resources immediately upon client disconnection.

Selection Recommendations

If the goal is to quickly deploy a general-purpose LLM service, vLLM and TGI are common choices. TGI emphasizes continuous batching, streaming, Prometheus metrics, and OpenTelemetry tracing; the vLLM ecosystem focuses on PagedAttention, scheduling, KV Cache management, and high-throughput serving. TensorRT-LLM is more oriented toward deep optimization on NVIDIA GPUs, with in-flight batching, paged attention, quantization, and multi-GPU inference capabilities better suited for performance-critical deployments.

When selecting, don’t just ask “which framework is fastest.” More relevant questions are:

Does my business prioritize TTFT or total throughput?
Is the request length distribution stable?
Do I need an OpenAI-compatible API?
Do I need multi-GPU / multi-node?
Can my team maintain a lower-level optimization stack like TensorRT-LLM?
Do I already have a monitoring system integrated with Prometheus / OpenTelemetry?

FAQ

Does Continuous Batching change model output?

Generally, no. It changes server-side scheduling, not model weights or sampling logic. However, if you also adjust sampling parameters, max output length, stop conditions, or framework version during deployment, outputs may still differ, so canary testing is still necessary.

How does it relate to PagedAttention?

They solve different problems but often appear together. Continuous Batching addresses request scheduling and GPU utilization; PagedAttention addresses KV Cache memory management. Without efficient KV Cache management, increasing running requests makes memory bottlenecks more likely.

Do low-traffic services need Continuous Batching?

Benefits are limited in low QPS scenarios because the GPU doesn’t have enough requests to batch. However, retaining this capability is still valuable: when traffic spikes or multiple tenants share the service, it can reduce GPU idle time and queuing jitter.