PagedAttention in Production: Boosting LLM Serving Throughput with Paged KV Cache

The Problem: LLM Serving Bottlenecks Are Often KV Cache, Not Weights

In online LLM inference, model weights are typically static. What changes dynamically with each request is the KV Cache. An autoregressive generation cycle has two phases:

• Prefill: Process the input prompt, compute Key/Value for each attention layer, and write to the KV Cache.
• Decode: Generate one or more new tokens per step, reusing the historical KV Cache to avoid recomputing the full context.

When context length, concurrent requests, and output length all grow, the KV Cache quickly becomes the dominant memory pressure. Worse, different requests have vastly different prompt and output lengths. Traditional contiguous memory allocation leads to two types of waste:

1. Reservation waste: To support the maximum sequence length, the system reserves KV space for each request up to the limit, but most requests never use it all.
2. Fragmentation waste: As requests enter, generate, finish, and release memory, small holes appear that are hard to reuse. Even with free memory remaining, you can’t fit enough active requests.

This directly limits batch size, hurting throughput, queue time, and Time To First Token (TTFT). This is where PagedAttention shines: it transforms the KV Cache from a “contiguous large array” into a paged block management system, similar to operating system virtual memory.

Core Principle: Splitting the KV Cache into Fixed-Size Pages

The core idea of PagedAttention is straightforward: don’t require a request’s KV Cache to reside in a contiguous physical memory region. Instead, split it into multiple fixed-size KV blocks.

Think of a request’s token sequence as a logical address space:

Request A logical tokens:
[0..15] [16..31] [32..47] [48..63] ...
   B0      B1       B2       B3

Each logical block is then mapped to a physical block in GPU memory via a block table:

Logical block table:
  B0 -> physical block 18
  B1 -> physical block 04
  B2 -> physical block 27
  B3 -> physical block 09

These physical blocks don’t need to be contiguous. When a new token is generated and the current block is full, a new block is allocated from the free block pool. When a request finishes, its blocks are returned to the pool for reuse by other requests.

This mechanism brings three direct benefits:

1. Reduced Memory Fragmentation

Traditional contiguous allocation requires finding a large enough contiguous space for variable-length sequences. PagedAttention only needs a few fixed-size free blocks, no longer relying on contiguous addresses, significantly mitigating external fragmentation.

2. Reduced Over-Reservation

The system no longer needs to reserve KV Cache for each request up to the maximum length upfront. Instead, blocks are allocated on-demand as the request actually generates tokens. This on-demand allocation better matches actual usage.

3. More Flexible Sharing and Reuse

When multiple requests share the same prefix, the underlying KV blocks can be reused or managed via reference counting. vLLM’s Automatic Prefix Caching is built on the foundation that KV blocks are identifiable, mappable, and reusable.

PagedAttention and Continuous Batching

In many engineering practices, PagedAttention and Continuous Batching are discussed together, but they operate at different levels.

Mechanism	Problem Solved	Core Action
PagedAttention	KV Cache memory management	Splits KV Cache into fixed-size blocks, mapped via a block table
Continuous Batching	Request scheduling	Dynamically combines new, in-progress, and finishing requests each decode iteration

Combined, they form a production-ready LLM Serving architecture:

Incoming requests
  -> Scheduler / continuous batching
    -> Prefill and decode iterations
      -> Paged KV block allocation
        -> Attention kernel reads block table
          -> Streaming response
            -> Block release / reuse

Without paged KV Cache, continuous batching is easily limited by memory fragmentation and reservation waste. Without continuous batching, PagedAttention only solves the “can it fit” problem, not the “can it run efficiently” problem.

Engineering Implementation: From Memory Model to Service Configuration

1. Confirm Your Workload Is a Good Fit

PagedAttention is most valuable for:

• Multi-tenant online inference services with highly variable request lengths.
• Chat/Agent scenarios with fluctuating input context and output lengths.
• Long-context tasks like document Q&A, code analysis, and report generation.
• Services requiring continuous streaming output and higher concurrency throughput.
• Cases where GPU memory is often exhausted by KV Cache, not just model weights.

If your service is primarily short prompts, short outputs, low concurrency, or offline batch processing, PagedAttention may still help, but the gains won’t be as dramatic as with long-context online services.

2. Understand the Impact of Block Size

PagedAttention typically uses a fixed number of tokens per KV block. A block size that’s too small increases block table and scheduling overhead. A block size that’s too large increases internal waste in the last block.

Don’t just look at peak throughput from a single benchmark. Also observe:

• Memory usage under different prompt length distributions.
• Block allocation and release frequency under different output lengths.
• Whether waiting requests decrease under high concurrency.
• Whether both TTFT and TPOT (Time Per Output Token) improve, or just throughput.

3. Use with Prefix Caching

PagedAttention itself is not prefix caching, but it provides a natural infrastructure for it. vLLM’s prefix caching identifies a KV block by a hash of its prefix tokens and current block tokens, allowing subsequent requests sharing the same prefix to reuse the cache.

For RAG, Agent, and applications with long system prompts, place stable content at the beginning of the prompt:

[system prompt / tool schema / policy / fixed examples]
[user-specific dynamic question]

This makes it easier to form stable prefixes and improve cache hit rates. Conversely, if every request starts with a timestamp, random trace ID, or dynamic user info, prefix cache hit rates will be severely degraded.

4. Synergy with Chunked Prefill

Long prompt prefill consumes significant compute and memory bandwidth in one shot. If you cram an extremely long context into a single prefill, it can squeeze out decode phases, causing streaming token jitter.

Some inference frameworks combine chunked prefill or similar mechanisms, splitting long contexts into multiple chunks and interleaving prefill chunks with decode requests. PagedAttention provides the KV block management foundation, while chunked prefill provides finer-grained scheduling control.

A Simplified Service Configuration Approach

Below is an example startup approach in the style of vLLM. Actual parameters should be determined by your model, GPU, framework version, and workload load testing.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching

Parameter breakdown:

Parameter	Purpose	Notes
max-model-len	Controls the maximum context length	Directly impacts the upper limit of KV Cache capacity
gpu-memory-utilization	Controls how the inference engine allocates memory for model weights, runtime, and KV Cache	Don’t set to 100%; leave a safety margin
enable-prefix-caching	Enables automatic prefix caching	Suitable for scenarios with stable prefix reuse

Before going live, load test with real request distributions, not just a single fixed-length prompt.

Production Monitoring: Don’t Just Watch GPU Utilization

The benefits of PagedAttention are primarily seen in memory utilization, queue length, and token latency. Therefore, monitoring should cover at least four categories.

1. Memory and KV Cache

Metric	Description
KV cache usage	Is KV Cache usage consistently near the limit?
Preemption / Eviction	Are request preemptions or cache evictions frequent?
Prefix cache hit rate	Is prefix caching actually hitting when enabled?
Block lifetime / idle time	Are there many blocks held long-term with low reuse?

2. Scheduling and Queues

Metric	Description
num_requests_running	Number of requests currently executing
num_requests_waiting	Number of requests waiting to be scheduled
Queue time	How long requests wait before entering model execution
Iteration tokens	Number of tokens processed per engine step

3. Latency

Metric	Description
TTFT (Time To First Token)	Time until the user sees the first token
TPOT / ITL (Inter-Token Latency)	Interval between generated tokens
E2E latency	Total end-to-end request time

If throughput improves but P95 TTFT worsens significantly, the scheduling policy may be overly aggressive in maximizing batch efficiency at the expense of interactive experience.

4. Quality and Stability

PagedAttention should not change model semantic output, but engineering-level scheduling, prefix caching, chunked prefill, and concurrency stress tests can still expose implementation bugs or edge cases. Before going live, prepare a fixed regression test set covering:

• Short Q&A
• Multi-turn conversations
• Long document summarization
• RAG with citation-style answers
• Tool calling or structured output
• High-concurrency streaming

Common Misconceptions

Misconception 1: PagedAttention Makes Every Single Request Faster

PagedAttention primarily optimizes concurrent throughput and memory utilization. Single-request latency may improve or remain similar. Its core value isn’t “make one request faster,” but “support more active requests under the same memory and reduce dynamic memory waste.”

Misconception 2: Only Looking at Average Latency, Ignoring P95/P99

Online services are most vulnerable to tail latency. PagedAttention with continuous batching can improve average throughput, but if scheduling parameters are poorly tuned, P95 TTFT or TPOT can still degrade. Production decisions should be based on P95/P99, queue length, and error rates.

Misconception 3: Prefix Caching Is Automatic Free Performance

Prefix caching relies on stable prefixes. If the beginning of the prompt has dynamic fields, cache hits can be destroyed. System prompts, tool schemas, examples, and retrieval templates should be as stable as possible and placed in the prefix region.

Misconception 4: Maxing Out Memory Is Optimal

KV Cache usage consistently near 100% is not a good sign. It means any request fluctuation can cause queuing, preemption, OOM, or aggressive cache eviction. Production environments should maintain a safety margin.

Misconception 5: Ignoring Version Differences

Frameworks like vLLM, TensorRT-LLM, and TGI are evolving rapidly. Paged KV cache, prefix caching, chunked prefill, metric names, and default parameters can change between versions. Before going live, pin your framework version and record a configuration snapshot.

Pre-Deployment Checklist

Workload Modeling

• Collect real distributions of prompt length, output length, concurrency, and QPS.
• Distinguish between Chat, RAG, Agent, batch generation, and long-document tasks.
• Record P50/P95/P99 input and output lengths separately.

Load Test Design

• Use real length distributions, not fixed short prompts.
• Test cold start, steady traffic, burst traffic, and long-output traffic separately.
• Compare results with and without prefix caching.
• Record throughput, TTFT, TPOT, queue time, and KV cache usage.

Configuration Governance

• Pin model version, inference framework version, and GPU model.
• Record max model length, memory utilization, and batch/token limits.
• Set up different routing or instance pools for long-context and short Q&A tasks.
• Prepare a one-click rollback plan to the previous inference service.

Monitoring and Alerts

At a minimum, configure the following alerts:

alerts:
  - name: HighKVCacheUsage
    condition: kv_cache_usage_perc > 0.90 for 5m
  - name: WaitingRequestsGrowing
    condition: num_requests_waiting p95 keeps increasing for 5m
  - name: TTFTRegression
    condition: p95_time_to_first_token increases by 30% over baseline
  - name: TPOTRegression
    condition: p95_inter_token_latency increases by 30% over baseline
  - name: PrefixCacheHitDrop
    condition: prefix_cache_hit_rate drops below expected baseline

Don’t copy thresholds directly; determine them based on your business baseline.

Conclusion

PagedAttention is not a simple “single-point speed-up switch.” It is a KV Cache virtual memory management mechanism designed for LLM Serving. By splitting the dynamically growing KV Cache into fixed-size blocks and using logical-to-physical mapping, it reduces memory fragmentation and over-reservation, allowing more active requests to be scheduled by continuous batching.

In production, PagedAttention works best when combined with continuous batching, prefix caching, chunked prefill, Prometheus metrics, and load testing with real length distributions. As long as monitoring and rollback mechanisms are in place, it is often an essential foundation for building high-throughput, long-context, cost-effective LLM inference services.

Key References

• Efficient Memory Management for Large Language Model Serving with PagedAttention, arXiv: https://arxiv.org/abs/2309.06180
• vLLM Automatic Prefix Caching Design: https://docs.vllm.ai/en/v0.9.2/design/automatic_prefix_caching.html
• vLLM Production Metrics: https://docs.vllm.ai/en/v0.20.0/usage/metrics/
• vLLM Metrics Design: https://docs.vllm.ai/en/stable/design/metrics/
• Hugging Face Text Generation Inference Documentation: https://huggingface.co/docs/text-generation-inference/en/index
• NVIDIA TensorRT-LLM GPT Attention / In-flight Batching / Paged KV Cache: https://github.com/NVIDIA/TensorRT-LLM
• vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention, arXiv: https://arxiv.org/abs/2405.04437