Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

Background: LLM Serving Bottlenecks Go Beyond Model Compute

In traditional online services, batching is intuitive: accumulate a group of requests, execute them together, and return the entire batch. This works naturally for image classification, embeddings, or standard RPC aggregation because each request has a relatively fixed execution length. However, LLM generation is autoregressive—the model first processes the prompt, then generates output tokens one by one until a stop condition is met or the maximum length is reached.

This introduces a direct problem: output lengths within the same batch often vary dramatically. One user might generate only 20 tokens, while another requests a long article producing 800 tokens. Under static batching, short requests, though completed early, are held back by long ones; new arriving requests must wait until the entire batch finishes before they can enter.

The ORCA paper attributed this issue to the multi-iteration nature of Transformer generation workloads and proposed iteration-level scheduling—scheduling at the granularity of decoding iterations rather than complete requests. Continuous Batching (also known as in-flight batching or iteration batching) is the engineering realization of this idea in production inference services. Its goal is not to change model quality, but to keep the GPU continuously producing useful tokens under high concurrency, reducing padding, waiting, and long-tail idle time.

Core Principle: From “Request-Level Batching” to “Iteration-Level Scheduling”

The Problem with Static Batching

The basic flow of static batching is as follows:

Batch A = [request_1, request_2, request_3]
run prefill
while not all requests finished:
    run one decode step for all requests in Batch A
return all results
then accept Batch B

This approach suffers from three typical types of waste:

Waste Type	Manifestation	Impact
Short requests blocked by long ones	Short requests finish early but remain in the batch lifecycle	Resource idle, degraded user experience
New requests cannot join promptly	Server must wait for the current batch to fully complete	Increased queuing time, worsened TTFT
Padding and resource holes	Uneven prompt/output lengths cause compute and memory imbalance	Low GPU utilization, reduced effective throughput

For interactive chat, AI agents, code completion, online customer service, and similar scenarios, these wastes directly inflate TTFT (Time To First Token) and tail latency.

How Continuous Batching Schedules

Continuous Batching reduces the scheduling granularity from “complete request” to “single decode iteration.” A typical flow is:

active_batch = []
loop:
    add_new_requests_if_capacity_allows(active_batch)
    run_prefill_for_new_requests_or_chunked_prefill()
    run_one_decode_iteration(active_batch)
    stream_new_tokens_to_clients()
    remove_finished_requests(active_batch)
    free_or_reuse_kv_cache_blocks()

The key change is: after each decoding round, the scheduler can remove completed requests from the batch and simultaneously add new ones. The TensorRT-LLM Batch Manager documentation explicitly describes in-flight batching: adding newly arrived requests at each iteration of the token generation loop and returning results immediately when a request reaches its end condition, rather than waiting for the entire batch to finish.

This type of scheduling typically needs to be co-designed with KV Cache management, PagedAttention, chunked prefill, and request queuing strategies. Otherwise, even though the batch is dynamically variable, memory, long prompt prefill, or scheduling policies can still become bottlenecks.

Why Continuous Batching Improves Throughput

1. Reduces Long-Tail Waste Within a Batch

LLM output lengths are unpredictable. In a static batch, a single long response can hold up the entire batch. Continuous Batching allows short requests to release their slots immediately upon completion, and new requests fill in, making the effective workload of the batch more stable.

2. Increases GPU Utilization

In online services, requests arrive continuously. Static batching either waits for more requests to form a batch (increasing queuing time) or runs with a small batch (lowering GPU utilization). Continuous Batching dynamically absorbs new requests at each iteration, making it easier to maintain a high number of running requests.

3. Reduces Padding and Waiting Costs

The TensorRT-LLM documentation describes traditional V1 batching as running requests in lockstep, padding to the maximum input/output length in the batch. InflightBatching dynamically incorporates new requests and returns them immediately upon completion, eliminating the need for uniform padding. This is the engineering value of Continuous Batching: reducing computation that does not produce useful tokens.

4. Amplifies Benefits with KV Cache Optimization

While Continuous Batching increases the number of concurrent requests, it also amplifies KV Cache memory pressure. Serving engines like vLLM, TGI, and TensorRT-LLM typically combine continuous batching with PagedAttention, KV cache block management, prefix caching, or chunked prefill. The Hugging Face TGI documentation lists continuous batching, streaming, Flash Attention, and Paged Attention as core capabilities for inference serving.

Engineering Deployment: Don’t Just Flip a Switch

Step 1: Identify Your Workload

Continuous Batching is best suited for:

• Continuously arriving request scenarios like online chat, code completion, AI agents, and customer service assistants
• Workloads with high variance in output length, where short and long requests are mixed
• Unstable GPU utilization with frequent fluctuations in the waiting queue and running batch
• Services requiring token streaming, where users care about TTFT and response fluency

Less suitable scenarios include: offline batch generation with controllable input/output lengths, very low request volumes (limited benefit), and dedicated low-latency services.

Step 2: Control Batch Limits and Token Budget

In production, it’s not advisable to control the batch solely by request count. A more reasonable approach is to monitor the following dimensions simultaneously:

Control Dimension	Description
Max concurrent sequences	Upper limit on requests participating in decode simultaneously
Max batch tokens	Upper limit on total tokens within a batch
Prefill token budget	Number of tokens a single prefill can handle
Decode token budget	Number of tokens a single decode can handle
Available GPU KV Cache blocks	Hard constraint at the memory level
Preemption/swap-out/recompute strategy	Whether to allow aggressive scheduling

When the scheduler is too aggressive, short-term throughput may increase, but tail latency will worsen, potentially triggering KV Cache pressure and request reordering. The TensorRT-LLM batch manager documentation notes that scheduling policies can favor maximizing utilization or adopt a more conservative no-evict strategy—the former pursues GPU throughput, while the latter prioritizes memory predictability.

Step 3: Handle the Prefill vs. Decode Conflict

LLM requests have two phases:

Phase	Characteristics	Bottleneck
Prefill	Processes the full input prompt, computes state before the first output token	Large matrix computation, compute-intensive
Decode	Generates output tokens one by one	KV Cache reads/writes, memory-intensive

If a long prompt’s prefill is inserted directly into a batch that is actively streaming, it can cause noticeable pauses in the tokens perceived by users. Therefore, many engines introduce chunked prefill, which processes long prompts in segments, allowing prefill and decode to coexist more smoothly. A 2025 Hugging Face technical article explains KV caching, chunked prefill, ragged batching, and dynamic scheduling together, emphasizing that combining them is key to maximizing throughput.

Step 4: Keep Degradation Strategies

Continuous Batching is a scheduling optimization, not a risk-free switch. At a minimum, retain the following during deployment:

• A configuration path to disable continuous batching
• A quick toggle to reduce max concurrent requests
• A strategy to limit max output tokens
• A separate queue or rate limiter for very long prompts
• Isolation between high-priority interactive requests and low-priority batch tasks

Common Misconceptions

Misconception 1: Only Looking at Tokens/s, Ignoring User Experience

Continuous Batching often improves overall tokens/s, but user experience also depends on TTFT and TPOT / ITL. If throughput gains come from continuously piling up the queue, users will experience slower first tokens, not a faster service.

Misconception 2: Treating Batch Size as the Only Tuning Parameter

The batch in LLM serving is not the fixed batch of traditional deep learning inference. More important are token budget, KV Cache blocks, the prefill/decode mix ratio, the distribution of request output lengths, and the scheduling policy. Tuning only max_batch_size can easily lead to misleading results.

Misconception 3: Ignoring the Interference of Long Prompts on Decode

The prefill computation for long-context requests is substantial. Without chunked prefill or queue isolation, it can cause noticeable stalls for requests that are actively streaming. This is especially critical for agent scenarios—tool call results, conversation history, and retrieved context can quickly inflate prompts.

Misconception 4: Believing Continuous Batching Automatically Reduces All Latency

Its primary function is to improve resource utilization and reduce waste within batches. Under low load, short outputs, single-user, or strict low-latency scenarios, the benefits are limited. Under high load, if scheduling is not handled properly, it can actually increase queuing time and tail latency.

Go-Live Checklist

Performance Baseline

Run three sets of baselines before going live:

1. Throughput, TTFT, TPOT, P95/P99 latency under static or default scheduling
2. The same metrics with Continuous Batching enabled
3. Stress tests under mixed scenarios: high concurrency, long prompts, long outputs, short outputs

Don’t just look at averages. The most common issues in LLM services are with P95/P99, long prompts, and queuing during peak hours.

Monitoring Metrics

Monitor at least the following dimensions:

• num_requests_running: Number of requests currently being executed by the model
• num_requests_waiting: Number of requests waiting to enter the scheduler
• KV-cache usage: GPU KV Cache usage ratio
• TTFT: Time from request entry to first token returned
• TPOT / ITL: Interval between output tokens or time to generate a single token
• prompt tokens/s and generation tokens/s: Differentiate prefill and decode load
• Timeout rate, cancellation rate, error rate
• Preemption, swap-out, recompute counts (if exposed by the serving engine)

The vLLM documentation notes that its metrics endpoint can expose production metrics like running requests, waiting requests, and KV-cache usage; its design documents also emphasize latency metrics like TTFT, TPOT, and prefill interval.

Alerting Recommendations

Production alerts should not rely solely on GPU utilization. A more practical combination is:

alerts:
  - name: high_waiting_queue
    condition: num_requests_waiting consistently above baseline
    action: Scale out, rate limit, or reduce low-priority traffic

  - name: kv_cache_pressure
    condition: gpu_kv_cache_usage near limit AND TTFT rising simultaneously
    action: Reduce max concurrent sequences, limit long contexts, check preemption

  - name: streaming_stall
    condition: TPOT or ITL P95 significantly elevated
    action: Check if long prefill is blocking decode, enable or reduce chunked prefill size

  - name: throughput_regression
    condition: tokens_per_second dropping but request volume not decreasing
    action: Check model version, quantization config, batch token budget, GPU errors, and cache hits

A Simplified Engineering Configuration Approach

Below is an abstract configuration example, not tied to any specific framework:

serving:
  scheduler:
    mode: continuous_batching
    max_running_requests: 128
    max_waiting_requests: 2048
    max_batch_tokens: 8192
    prefill_chunk_size: 1024
    priority_policy: interactive_first
  limits:
    max_input_tokens: 16384
    max_output_tokens: 2048
    reject_or_queue_when_overloaded: queue_with_timeout
kv_cache:
  block_manager: paged
  max_gpu_cache_usage: 0.90
  enable_prefix_cache: true
observability:
  export_prometheus_metrics: true
  track_ttft: true
  track_tpot: true
  track_queue_time: true
  track_kv_cache_usage: true

The key point of such a configuration is not the specific values, but viewing scheduling, token budget, KV Cache, priority, and monitoring under a single unified perspective.

Suitable and Unsuitable Scenarios

✅ More Suitable

• Multi-user online chat services
• API services with token streaming
• Continuous request arrival with uneven output length distribution
• GPU cost-sensitive environments aiming to increase throughput
• Need to stably control queuing time under high concurrency

⚠️ Requires Caution

• Dedicated services with extremely low latency and low concurrency
• Offline batch processing with fully controllable output lengths
• Environments with a very high proportion of long contexts but no chunked prefill or queue isolation
• Environments lacking monitoring for TTFT, TPOT, KV Cache, and waiting queue

FAQ

Does Continuous Batching affect model output quality?

Theoretically, it is a serving scheduling optimization that does not change model weights, sampling parameters, or logits computation logic, so it should not alter output quality. However, in engineering implementation, you must still verify edge cases like stop sequences, random seeds, streaming order, request cancellation, and timeout retries.

What is its relationship with PagedAttention?

Continuous Batching solves the “scheduling problem,” while PagedAttention solves the “KV Cache memory management problem.” They often appear together because dynamic scheduling increases the number of concurrent sequences, and the higher the concurrency, the more critical KV Cache management becomes.

Why might TTFT actually worsen under high throughput?

If the scheduler continuously fills the batch to maximize throughput, new requests may queue longer in the waiting queue. Alternatively, too many long prompt prefills can disrupt the decode iteration rhythm. In such cases, overall tokens/s may look good, but the interactive experience degrades.

Conclusion

Continuous Batching is a key optimization that moves LLM Serving from “batch-and-execute” to “continuous scheduling.” By using iteration-level scheduling, it allows completed requests to exit promptly and new requests to enter, reducing the long-tail waiting and GPU idle time inherent in static batching.

However, it is not an isolated switch. True production deployment requires simultaneously addressing KV Cache management, chunked prefill, token budget, request priority, monitoring alerts, and degradation strategies. For engineering teams, the safest go-live approach is: first establish a static baseline, then gradually roll out using a canary approach, continuously observing TTFT, TPOT, waiting queue, and KV-cache pressure. Only when throughput improvements do not come at the cost of critical latency metrics has Continuous Batching truly delivered its value.

Key References

• ORCA: A Distributed Serving System for Transformer-Based Generative Models, USENIX OSDI 2022 — https://www.usenix.org/conference/osdi22/presentation/yu
• Hugging Face Text Generation Inference documentation — https://huggingface.co/docs/text-generation-inference/en/index
• Hugging Face Blog: Continuous batching from first principles, 2025-11-25 — https://huggingface.co/blog/continuous_batching
• NVIDIA TensorRT-LLM Overview — https://nvidia.github.io/TensorRT-LLM/overview.html
• TensorRT-LLM Batch Manager documentation — https://github.com/nyunAI/TensorRT-LLM/blob/main/docs/source/batch_manager.md
• vLLM Metrics documentation — https://docs.vllm.ai/en/stable/design/metrics/
• vLLM Production Metrics documentation — https://docs.vllm.ai/en/v0.20.0/usage/metrics/