Continuous Batching in Production: Keeping LLM Services GPU-Saturated Under High Concurrency

Background

When many teams first deploy an LLM service, they often carry over the batching approach from traditional inference services: collect multiple requests into a batch and feed them all into the model at once. This works well for relatively short, stateless models like image classification, text classification, or embedding. However, in the context of autoregressive LLM generation, three problems quickly emerge.

First, LLM generation isn’t a single forward pass; it consists of two phases: prefill + decode. Prefill processes the full input prompt, while decode generates output tokens one by one. Second, different requests have vastly different input and output lengths. Within the same batch, some requests may finish quickly while others need to generate hundreds more tokens. Third, each request holds a growing KV Cache. GPU memory isn’t just consumed by model weights; it’s also continuously consumed by the dynamic state of each request.

If you stick with request-level static batching, you’ll encounter classic head-of-line blocking: short requests must wait for long requests to finish, and new requests must wait for the entire current batch to complete. The GPU appears to be working, but throughput, tail latency, and memory utilization are all unstable.

The core goal of Continuous Batching is to move the batching granularity from the “request level” down to the “iteration level” or “token level.” After each decode iteration, completed requests are removed from the execution set, and new requests from the waiting queue are added. This keeps the GPU processing valid tokens as continuously as possible, rather than waiting for an entire old batch to finish naturally.

Core Principle: From Request-Level Batching to Iteration-Level Scheduling

Traditional static batching looks like this:

collect requests -> build batch -> run until every request finishes -> return all results

In LLMs, this approach causes short-output requests to be held up by long-output ones. Continuous batching changes this to:

while server is running:
    admit new requests if token and memory budget allow
    run one prefill or decode step for selected sequences
    stream generated tokens
    remove finished sequences
    keep unfinished sequences in the active set

This change might seem like just a scheduling optimization, but it has a significant impact on production services:

• New requests don’t have to wait for the entire batch to finish; they only need to wait for the next scheduling point.
• Completed requests can release resources promptly, reducing wasted capacity.
• GPU execution is more continuous, and throughput is generally more stable than with pure request-level batching.
• Queue policies can be more granular, for example, separating short requests, long prompts, and low-priority background tasks.

The ORCA paper refers to this approach as iteration-level scheduling, where the scheduler drives the execution engine at the granularity of model iterations, rather than scheduling at the granularity of complete requests.

Different Resource Profiles for Prefill and Decode

Continuous batching can’t just look at the “number of requests”; it must distinguish between two types of tokens:

Type	Characteristics	Bottleneck
Prefill token	From the input prompt, highly parallelizable, consumes significant compute and memory bandwidth in one shot	Compute-bound
Decode token	Typically one or a few tokens generated per active sequence per round	Memory bandwidth-bound

Therefore, production schedulers typically don’t just set a max_batch_size. They introduce budgets that are more tailored to LLMs:

max_num_seqs: 128            # Maximum number of concurrently active sequences
max_num_batched_tokens: 8192 # Total number of tokens allowed in a single scheduling step
max_model_len: 32768         # Maximum context length per request
kv_cache_blocks: 40000       # Number of allocatable KV cache blocks
prefill_chunk_size: 2048     # Whether to chunk long prompts before execution

A common mistake is to only pursue a larger batch size. For LLMs, a larger batch isn’t always better. An excessively large batch can lead to longer prefill queues, decode iterations being interrupted by long prompts, and KV Cache approaching its watermark. The result might be higher throughput, but users will experience significantly worse time-to-first-token (TTFT) and p99 latency.

KV Cache: The True Constraint of Continuous Batching

GPU memory in an LLM service isn’t just for model weights. During generation, each request saves the key/value states of all historical tokens, known as the KV Cache. The more requests, the longer the context, and the longer the output, the larger the KV Cache footprint.

This is why continuous batching is often discussed alongside PagedAttention, KV block management, prefix caching, and KV cache quantization. For the scheduler to continuously admit new requests, KV Cache management must be granular enough to avoid premature OOM due to fragmentation, excessive reservation, or excessive copying.

The key idea of PagedAttention is to divide each sequence’s KV Cache into fixed-size blocks. These blocks are logically contiguous but can be physically non-contiguous. This allows for on-demand allocation, reuse, and release of KV blocks, reducing fragmentation and allowing more active requests to coexist on the GPU.

Architecture Flow

A production-grade continuous batching service can typically be broken down into the following layers.

1. API Router

Responsible for receiving requests, validating parameters, setting timeouts, and tagging tenants, task types, and priorities. It shouldn’t just be a simple forwarder; it needs to provide the downstream scheduler with sufficient metadata.

It’s recommended to record at least:

• Tenant or business line
• Model name and version
• Number of input tokens
• Maximum number of output tokens
• Whether streaming output is requested
• Priority
• SLA type
• Acceptable timeout

2. Admission Control

Admission control determines whether a request can enter the waiting queue. It needs to consider the number of requests, token counts, KV Cache budget, and current queue wait time simultaneously.

def admit(request, state):
    estimated_total_tokens = request.input_tokens + request.max_output_tokens
    if state.waiting_requests > LIMIT_WAITING_REQUESTS:
        return Reject("queue is full")
    if state.active_sequences >= LIMIT_ACTIVE_SEQS:
        return Wait()
    if state.kv_blocks_free < estimate_blocks(estimated_total_tokens):
        return Wait()
    if request.deadline_ms < state.estimated_wait_ms:
        return Reject("deadline cannot be met")
    return Accept()

The key point here isn’t the complexity of the formula, but to avoid the service “unconditionally accepting requests.” Unconditionally accepting requests only shifts the pressure to the queue, ultimately manifesting as excessively long TTFT, batch timeouts, and avalanche retries.

3. Prefill Queue and Decode Queue

Prefill and decode should be observed and scheduled separately. If a long prompt’s prefill consumes the entire scheduling budget in one go, it can block ongoing streaming decode requests, making the user feel like the output has suddenly stalled.

Common strategies include:

• Chunked prefill: Splits very long inputs into multiple segments for scheduling, preventing a single block of decode.
• Decode-first or mixed policy: Prioritizes stable decode output in interactive scenarios.
• Long prompt lane: Routes requests like long document summarization or codebase analysis to a separate queue or dedicated instance.
• Background lane: Batch processing tasks, offline generation, and low-priority tasks should not be mixed with online chat requests.

4. Token Scheduler

The scheduler selects a set of “tokens to execute next” in each round. Its input isn’t a set of complete requests, but a set of candidate sequences and budget constraints.

while True:
    release_finished_sequences()
    update_kv_cache_watermark()
    candidates = []

    # 1. keep active decode streams moving
    candidates += pick_decode_sequences(
        max_sequences=decode_seq_budget,
        priority_policy="deadline_then_fair_share",
    )

    # 2. fill remaining token budget with prefill chunks
    candidates += pick_prefill_chunks(
        max_tokens=remaining_token_budget(),
        chunk_size=prefill_chunk_size,
    )

    # 3. execute one iteration
    outputs = model_executor.forward(candidates)

    # 4. stream tokens and update sequence state
    publish_outputs(outputs)
    update_sequence_state(outputs)

The key point is: the working set is re-selected in every round. This allows the system to adapt quickly when requests complete, time out, are cancelled, are preempted, or when KV Cache pressure changes, rather than being locked into an old batch.

5. KV Cache Manager

The KV Cache manager is responsible for allocating, reusing, releasing, and monitoring KV blocks. It needs to answer at least four questions:

1. How many new sequences can still be admitted?
2. Will long-context requests crowd out short online requests?
3. Are KV blocks promptly released when a request is cancelled?
4. Will prefix caching, beam search, or multi-candidate generation share or copy the cache?

In production, it’s recommended to make the KV Cache watermark a first-class input to the scheduler, rather than just waiting for an OOM and restarting the process.

Engineering Deployment Essentials

Don’t Just Configure Max Concurrency

Many services only configure max_concurrent_requests, but continuous batching truly requires multi-dimensional budgets: number of requests, number of sequences, prefill tokens, decode tokens, KV blocks, queue wait time, and SLA deadlines.

A recommended initial configuration:

serving:
  max_active_sequences: 128
  max_waiting_requests: 512
  max_batched_tokens: 8192
  max_model_len: 32768
  prefill_chunk_size: 2048
  queue_timeout_ms: 30000
  stream_idle_timeout_ms: 10000
  kv_cache_high_watermark: 0.88
  kv_cache_reject_watermark: 0.95

These values cannot be copied directly. The optimal values will vary depending on the model, GPU, quantization method, context length, and traffic distribution. You must stress-test with real prompt length distributions and output length distributions before going live.

Separate Queues for Different Tasks

The same model service often handles multiple types of requests simultaneously: chat, RAG Q&A, long document summarization, code generation, agent tool calls, and offline batch generation. Their token distributions and user experience goals are completely different.

It’s recommended to split them into at least three categories:

Queue Type	Typical Scenario	Key Focus
interactive	Online chat, customer service, Copilot	TTFT, streaming stability, p95
long-context	Long document summarization, codebase analysis	Prefill cost, KV Cache pressure
batch	Offline generation, bulk rewriting, data annotation	Throughput, cost

Different queues can share model weights, but they should not share the exact same scheduling policy.

Monitor TTFT and TPOT Separately

LLM latency cannot be viewed as just total time. It should be broken down into at least:

Metric	Meaning
TTFT (Time To First Token)	Time the user waits for the first token
TPOT (Time Per Output Token)	Average time per token during the generation phase
E2E latency	Total time from request entry to completion
queue delay	Time from request entry into the queue to start of execution
prefill time	Time to process the input prompt
decode time	Time to generate output tokens

When continuous batching is misconfigured, a common symptom is that tokens/s looks good, but TTFT or p99 latency has significantly worsened. Looking only at throughput can mask user experience issues.

Handle Cancellation, Timeouts, and Client Disconnects

In streaming output scenarios, it’s common for users to close their pages, for network connections to drop, or for upstream services to time out. If the server doesn’t promptly cancel the request, the KV Cache will continue to be occupied, and decode will continue to waste GPU resources.

It’s recommended to require:

• The API layer can detect client disconnects.
• The scheduler can remove cancelled requests from the waiting and active sets.
• KV Cache blocks can be released promptly.
• Cancelled requests are tracked in metrics, not silently dropped.
• Upstream timeouts are consistent with the model service timeout to avoid “the client leaves early, but the GPU keeps running.”

Stress-Test with Real Traffic, Not Just Empty Prompts

The effectiveness of continuous batching is highly dependent on the traffic pattern. Stress test data should at least cover:

• Input token distribution: short prompts, medium prompts, long prompts.
• Output token distribution: short answers, long answers, early stops.
• Concurrency patterns: burst traffic, steady traffic, multi-tenant mixing.
• Streaming ratio: SSE/WebSocket vs. non-streaming requests.
• Cancellation ratio: user interruptions, upstream timeouts, active generation stops.
• RAG scenarios: variations in context length and number of citations.

If you only test throughput with fixed-length prompts, you are likely to get overly optimistic results.

Applicable Scenarios

Scenarios suitable for continuous batching:

• High-concurrency streaming generation like online chat, customer service, and code assistants.
• RAG Q&A, where input lengths vary significantly and output lengths are unpredictable.
• Multi-tenant API services that need to balance throughput, isolation, and SLA.
• Private LLM gateways that need to serve multiple business lines on limited GPUs.
• Scenarios where offline batch processing and online requests coexist but can be isolated via queues and priorities.

Scenarios less suitable:

• Very low request volume where the GPU is not saturated.
• All requests have very consistent lengths, and static batching is already stable enough.
• Extreme pursuit of minimum latency for a single request, with a willingness to sacrifice throughput.
• Business requirements that cannot tolerate the latency jitter introduced by queue waiting and dynamic scheduling.

Common Misconceptions

Misconception 1: Larger Batch is Always Better

A larger batch usually increases throughput, but it can also increase queuing time and tail latency. The production goal isn’t simply to maximize tokens/s, but to maximize effective throughput within SLA constraints.

Misconception 2: Continuous Batching Automatically Solves OOM

Continuous batching is just a scheduling strategy; it cannot replace KV Cache management. Without KV block budgets, high-watermark protection, and a cancellation/release mechanism, the system can still OOM under long-context traffic.

Misconception 3: Only Look at Average Latency

Average latency masks the long tail. Online LLM services should focus more on p95, p99, TTFT, TPOT, and timeout rates.

Misconception 4: Prefill and Decode Can Be Mixed and Scheduled Arbitrarily

Long prefill blocks decode, and decode jitter affects the streaming experience. High-quality services typically require chunked prefill, decode-first, or multi-queue strategies.

Misconception 5: All Business Uses a Single Queue

Long document summarization, chat, agent tool calls, and offline generation have different resource profiles. Sharing a single queue will pollute each other’s SLAs.

Go-Live Checklist

Capacity and Configuration

• Measured real input token and output token distributions.
• Set maximum active sequences, maximum waiting requests, and maximum batched tokens.
• Set max model length to prevent a single request from consuming infinite KV Cache.
• Set KV Cache high watermark and reject watermark.
• Decided whether to enable chunked prefill.

Scheduling and Isolation

• Separated business queues for interactive, long-context, and batch tasks.
• Set priority, deadline, or fair scheduling policies.
• Handled request cancellation, client disconnects, and upstream timeouts.
• Verified that long prompts do not significantly block online decode.

Monitoring and Alerting

• Monitoring TTFT, TPOT, E2E latency, and queue delay.
• Monitoring prefill tokens/s, decode tokens/s, and GPU utilization.
• Monitoring KV Cache block usage, release speed, and OOM events.
• Logging request cancellation rate, timeout rate, and rejection rate.
• Metrics broken down by queue, model, tenant, and API endpoint.

Gradual Rollout and Rollback

• Support for disabling continuous batching by model, route, or tenant.
• Prepared fallback to more conservative concurrency and token budgets.
• Prepared rate-limiting and queuing strategies for burst traffic.
• Validated p95/p99 with real traffic replay.
• Prepared scheduling logs and metric snapshots for post-incident reviews.

FAQ

Should I implement continuous batching from scratch?

Generally, it’s not recommended to start from scratch. The complexity of the scheduler, KV Cache management, CUDA kernels, streaming, cancellation/release, and distributed parallelism is very high. Most teams are better off configuring, wrapping, and monitoring mature inference engines like vLLM, SGLang, TGI, or TensorRT-LLM.

Are continuous batching and PagedAttention the same thing?

No. Continuous batching is a scheduling strategy, while PagedAttention is a method for KV Cache memory management and attention execution. They often appear together because continuous batching needs to accommodate more active sequences, which amplifies the pressure on KV Cache management.

Why does throughput increase after enabling it, but user experience gets worse?

A common reason is that the scheduling goal only optimizes for throughput, without incorporating constraints for TTFT, decode stability, queue waiting, and long-prompt interference. You need to re-separate queues, limit prefill, set high-watermark protection, and use p95/p99 as go-live thresholds.

References

1. ORCA: A Distributed Serving System for Transformer-Based Generative Models, USENIX OSDI 2022 — https://www.usenix.org/conference/osdi22/presentation/yu
2. Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023 — https://arxiv.org/abs/2309.06180
3. vLLM Documentation — https://docs.vllm.ai/en/latest/
4. Hugging Face Text Generation Inference Documentation — https://huggingface.co/docs/text-generation-inference/en/index
5. Hugging Face TGI Architecture — https://huggingface.co/docs/text-generation-inference/en/architecture
6. NVIDIA Triton Dynamic Batching & Concurrent Model Execution — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tutorials/Conceptual_Guide/Part_2-improving_resource_utilization/README.html
7. Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving — https://arxiv.org/abs/2406.13511