Background: The Bottleneck in Inference Services Is Often Not “Can the Model Compute?”
When optimizing LLM services for the first time, many people focus on GPU compute power, quantization, FlashAttention, or model size. But in real online services, a more easily underestimated problem is: requests have different lengths, end at different times, and KV Cache grows dynamically. If memory management is coarse, the number of requests that can fit on the GPU drops, throughput suffers, and queue times increase.
This is why vLLM and PagedAttention quickly became core technologies in mainstream inference engines. The PagedAttention paper points out that LLM services need to batch enough requests to achieve high throughput, but each request’s KV Cache is large, grows during generation, and is released upon completion. If managed poorly, memory fragmentation and redundant storage limit batch size. The paper further concludes that vLLM, through PagedAttention, achieves 2-4x throughput improvements over systems like FasterTransformer and Orca in test scenarios, while maintaining similar latency.
In other words, the key problem for LLM inference services isn’t “how to compute a single request as fast as possible,” but “how to stably co-locate many requests of varying lengths and lifetimes within limited GPU memory.”
Understanding KV Cache: Why It Eats Inference Throughput
Every time an autoregressive model generates a token, it needs to attend to all previous tokens. To avoid recomputing the Key and Value of historical tokens at each step, the inference engine caches them — this is the KV Cache.
After a request enters the model, it typically goes through two phases:
Prefill: Process the Input Context in One Shot
After the user provides a prompt, document, chat history, or tool results, the model must first pass these input tokens through the Transformer to generate the initial KV Cache. This phase is compute-heavy and resembles a “large matrix computation.”
Decode: Generate Tokens One by One
After the first token, the model generates one (or a few) tokens at a time, but each step requires reading the historical KV Cache. At this point, the task becomes memory-bound: the GPU isn’t limited by compute, but by constantly reading/writing KV Cache, scheduling requests, and moving data.
The problem is that different requests have different input lengths, output lengths, and end times. If the system pre-allocates a large contiguous block of VRAM for each request, short requests waste space, and long requests trigger expansion and fragmentation. The PagedAttention paper likens this to virtual memory management in operating systems: requests are like processes, tokens are like bytes, and KV Cache blocks are like pages.
The Core of PagedAttention: From “Contiguous Chunks” to “Paged Blocks”
Traditional approaches manage a request’s KV Cache as a contiguous tensor. This is simple to implement but unfriendly to dynamic requests:
- The maximum request length is uncertain, so reserving too much is wasteful.
- Actual output lengths vary; releasing short requests’ memory creates fragmentation.
- In scenarios like beam search, parallel sampling, or shared system prompts, identical prefixes are hard to reuse.
PagedAttention divides each request’s KV Cache into fixed-size blocks. Logically, the request still sees a contiguous token sequence; physically, these blocks can be scattered across different VRAM locations, managed by a block table that maps logical to physical addresses.
This yields three direct benefits:
1. On-Demand Allocation, Reducing Internal Waste
Allocation happens as the request generates tokens. There’s no need to reserve full space upfront based on maximum length. This is especially important for chat, Agent, and RAG scenarios where output length is unpredictable.
2. Fixed Block Size, Mitigating External Fragmentation
All blocks are the same size. When released, they return to a pool for reuse by other requests. Compared to “large contiguous VRAM,” fixed-block management is closer to a memory pool, making it easier for the scheduler to determine how many more requests can fit.
3. Block-Level Sharing, Enabling Prefix Reuse
If multiple requests share the same prompt, system prompt, document beginning, or multi-turn chat history, they can reference the same KV blocks at the underlying level. Copy-on-write is used when modifications are needed, avoiding redundant storage.
This is why PagedAttention isn’t just an attention kernel optimization — it’s a server-side memory management problem.
Continuous Batching: Why Ordinary Batching Isn’t Enough
Many services initially batch multiple requests together to boost throughput. But ordinary batching has a classic problem: if one request in the batch is slow, it holds up the others; new requests must wait for the next batch.
The TensorRT-LLM documentation describes in-flight batching as continuous batching or iteration-level batching: context-phase sequences can be processed alongside generation-phase sequences, allowing better interleaving of requests, lower latency, and higher GPU utilization. It also emphasizes that input tensors must be packed for efficiency, avoiding padding single-token generation requests to the maximum input length.
This is critical for online systems, where real requests are a mix of lengths:
- Some ask a single question.
- Some bring tens of thousands of document tokens for RAG.
- Some generate only 20 tokens and stop.
- Some require long-form output with thousands of tokens.
The goal of continuous batching is for the scheduler to re-evaluate at each step: which requests are still decoding, which can enter prefill, which have finished and can release their KV Cache, and which new requests can be inserted in the gaps.
Chunked Prefill: Don’t Let Long Prompts Consume the Entire Budget
In long-context scenarios, prefill can easily consume the entire token budget in one shot, causing short requests to queue. The TensorRT-LLM documentation mentions chunked context, where the context is divided into chunks. This allows context chunks to be batched alongside generation-phase tokens, improving overall throughput and removing some input length constraints.
Here’s a practical breakdown:
| Strategy | Behavior | Impact |
|---|---|---|
| No chunking | Long document prefill consumes the entire budget → short requests wait | TTFT jitter |
| With chunking | Long doc chunk 1 + short request decode → long doc chunk 2 + new request prefill → long doc chunk 3 + other request decode | More stable TTFT |
The benefit isn’t just throughput — it’s also tail latency stability. If your service has an SLA, such as a maximum TTFT threshold, chunked prefill is often more controllable than simply increasing max_num_tokens.
Prefix Caching: Don’t Recompute the Same Prefix
vLLM’s Automatic Prefix Caching (APC) caches the KV Cache of existing queries. If a new query shares the same prefix as a cached one, the corresponding KV Cache is reused, skipping computation for the shared part. The documentation lists typical benefit scenarios: users repeatedly querying the same long document, and multi-turn conversations repeatedly carrying the same chat history.
This optimization is particularly suitable for:
Long-Document Q&A
Users ask consecutive questions about the same contract, annual report, API documentation, or requirements document. The document content is identical, but the questions differ. Without prefix caching, the long document must be prefilled each time; with it, only the document’s KV Cache is reused.
Agent Fixed System Prompts
Many Agents have long system prompts, tool descriptions, output protocols, and safety rules. If these are identical each time, they serve as a stable prefix.
Multi-Turn Conversations
Multi-turn sessions repeatedly place historical messages into the context. Earlier turns are stable, while new messages are appended. As long as the request structure is stable, historical prefixes can be reused.
However, prefix caching isn’t free. It requires cache space, hit-rate statistics, eviction policies, and multi-tenant isolation. If request prefixes are frequently disrupted by timestamps, random IDs, or dynamic sorting fields, the hit rate will be poor.
SGLang’s RadixAttention: From “Same Prefix” to “Prefix Tree Reuse”
The RadixAttention proposed in the SGLang paper also revolves around KV Cache reuse. It organizes reusable KV Cache across multiple generation calls into a radix tree, which is well-suited for complex LLM programs, multi-turn interactions, few-shot learning, RAG, and JSON decoding. The paper’s abstract notes that SGLang’s runtime, including RadixAttention and compressed FSM, achieves up to 6.4x throughput improvements over contemporary systems in various language and multimodal tasks.
Compared to simple “exact full-prefix matching,” a radix tree provides structured prefix indexing:
Common system prompt
├── Tool description A
│ ├── User question 1
│ └── User question 2
└── Tool description B
├── User question 3
└── User question 4If your application uses fixed templates with dynamic variables, or if multiple tasks share a large amount of initial content, RadixAttention offers more potential than simply checking if the entire prompt is identical.
New Developments in 2026: PagedAttention Expands to More Hardware and Scheduling Strategies
PagedAttention was initially discussed primarily in the context of GPU serving engines, but recent work has extended it to TPUs, FlexAttention, and commodity GPU scheduling.
- The Ragged Paged Attention paper (April 2026) targets TPU scenarios, proposing high-performance TPU attention kernels based on Pallas and Mosaic, and notes its integration as a foundation for vLLM and SGLang’s TPU backends.
- PersistentKV (June 2026) focuses on page-aware decode scheduling, emphasizing that in long-context decoding, throughput is determined not just by individual attention kernels, but by how pages, KV-head groups, sequence splits, and workqueues are organized.
This indicates a trend: the competitive edge in LLM Serving is shifting from “having PagedAttention” to “co-designing KV Cache, scheduler, hardware backend, and prefix reuse.”
Engineering Implementation: Don’t Just Flip Switches — Build an Observability Loop
If you’re planning to implement PagedAttention, continuous batching, and prefix caching in your system, follow this order.
Step 1: Quantify Request Patterns
At a minimum, track these metrics:
| Metric Category | Key Metrics |
|---|---|
| Token distribution | prompt_tokens p50 / p90 / p99, output_tokens p50 / p90 / p99 |
| Latency | TTFT p50 / p90 / p99, TPOT p50 / p90 / p99 |
| Concurrency & cancellations | Concurrent request count, request cancellation rate |
| Cache | Prefix cache hit rate |
| Memory | KV cache memory usage, GPU memory fragmentation or block pool utilization |
Without these metrics, tuning becomes guesswork.
Step 2: Separate Prefill and Decode Pressure
- If TTFT is high, it may be due to prefill queuing, long prompts blocking, or an improperly set batch token budget.
- If TPOT is high, it may be due to decode-phase bandwidth, KV Cache reads, too few active sequences in the batch, or an unsuitable attention kernel.
Analyzing prefill and decode separately is far more effective than looking at “average response time” alone.
Step 3: Route Different Workloads to Different Pools
Don’t throw all requests into the same service pool. Split them by request pattern:
| Pool Type | Characteristics |
|---|---|
| Short Q&A pool | Low latency, small context |
| Long document pool | High token budget, chunked prefill |
| Agent pool | High prefix reuse, enable prefix caching |
| Batch processing pool | Throughput-first, relaxed latency |
This is typically more stable than tuning a single global max_batch_size.
Step 4: Standardize Prompt Templates to Boost Cache Hits
To benefit from prefix caching, the application layer must also cooperate:
- Fix the order of system prompts.
- Avoid placing timestamps, traceIds, or random nonces at the prompt prefix.
- Keep RAG document ordering stable.
- Use fixed version numbers for tool descriptions; don’t concatenate different descriptions each time.
- Place dynamic fields as late as possible.
Prefix caching isn’t a problem the model layer can solve alone — it requires prompt layout engineering at the application layer.
Step 5: Plan for Eviction and Isolation Policies
More caching doesn’t always mean better results. You must consider:
- Whether multi-tenant cache sharing is allowed.
- Whether KV Cache for sensitive documents needs isolation.
- Whether cache eviction is LRU, cost-based, or hit-prediction-based.
- Whether long-document caches will crowd out high-frequency short prefixes.
- Whether all caches become invalid after model or tokenizer version changes.
Addressing these issues post-deployment is much harder.
Common Misconceptions
Misconception 1: Using vLLM Automatically Guarantees High Throughput
vLLM provides excellent underlying capabilities, but actual throughput depends on model size, request length distribution, batch token budget, VRAM size, concurrency, whether prefix caching is enabled, whether chunked prefill is used, and whether application-layer prompts are stable.
Misconception 2: A Larger max_batch_size Is Always Better
An excessively large batch parameter may increase throughput ceilings, but it can also increase queuing and time-to-first-token. The TensorRT-LLM documentation also emphasizes that max_num_tokens must balance throughput, GPU utilization, TTFT, and TPOT.
Misconception 3: Long-Context Problems Can Only Be Solved with More VRAM
More VRAM can alleviate the problem, but it can’t replace scheduling. What long-context scenarios truly need are: paged KV Cache, chunked prefill, prefix caching, proper queue isolation, and monitoring.
Misconception 4: Prefix Caching Automatically Solves All Redundant Computation
If dynamic fields are inserted at the beginning of the prompt each time, cache hits will be broken. The application layer must stabilize the prompt structure for the underlying cache to be effective.
Go-Live Checklist
- Measured prompt/output token distribution
- Separated TTFT and TPOT metrics
- Monitored KV Cache block utilization
- Enabled or evaluated PagedAttention / paged KV cache
- Evaluated chunked prefill for long-context scenarios
- Evaluated prefix caching for multi-turn conversations and long documents
- Checked prompt prefix stability
- Set up different business pools or routing strategies
- Stress-tested p50/p90/p99, not just averages
- Defined cache eviction, tenant isolation, and model upgrade invalidation strategies
Conclusion: LLM Service Optimization Is a System Engineering Problem of “Compute + Memory + Scheduling”
The value of PagedAttention isn’t a single trick; it’s that it makes the KV Cache management problem in LLM services explicit. Continuous batching addresses the dynamic entry and exit of requests, chunked prefill addresses long prompts monopolizing the budget, and prefix caching and RadixAttention address redundant context computation.
When a business enters the stage of multi-turn conversations, long-document Q&A, Agent tool calls, and high-concurrency serving, what truly determines cost and experience is often not “a slightly smaller model” or “a slightly more powerful GPU,” but whether the inference engine can organize KV Cache, request scheduling, cache reuse, and hardware backends into a stable system.
If treated as just a parameter toggle, the benefits will be limited. If treated as part of the service architecture, technologies like PagedAttention will truly deliver their value.
Key References
- vLLM / PagedAttention paper: Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180
- vLLM documentation: Automatic Prefix Caching — https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/
- NVIDIA TensorRT-LLM documentation: Paged Attention, IFB, and Request Scheduling — https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html
- SGLang paper: SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
- Ragged Paged Attention: A High-Performance and Flexible LLM Inference Kernel for TPU — https://arxiv.org/abs/2604.15464
- PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs — https://arxiv.org/abs/2606.26666