KV Cache Reuse in Practice: Prefix Caching, Offloading, and Long-Context Cost Management

Background: Why Long-Context Services Stall on KV Cache

Large model inference typically splits into two phases: prefill and decode.

• prefill: The model processes the entire input at once—user prompt, system prompt, tool descriptions, RAG documents, conversation history—and generates Key/Value states for each Attention layer.
• decode: The model generates output tokens one by one, reading the previously saved Key/Value states at each step.

This set of Key/Value states is the KV Cache. Its value is straightforward: already-computed history tokens don’t need to be recomputed at every generation step. But it also introduces an engineering challenge—the longer the context, the higher the concurrency, and the more model layers, the more GPU memory the KV Cache consumes.

In RAG, Agent, code assistant, long-document Q&A, and multi-turn customer service scenarios, requests often contain large amounts of repeated content. For example, the same technical document is queried by many users, the same system prompt is carried by every request, and the same Agent tool description is included in every context. If each request performs a full prefill from scratch, the same prefix is recomputed repeatedly, wasting both GPU time and memory bandwidth.

Therefore, production-grade LLM serving must go beyond model weight quantization and batch size optimization; KV Cache reuse must be a first-class optimization target.

Core Problem: It’s Not “More Cache is Better,” but “Is the Cache Worth Restoring?”

KV Cache reuse sounds like ordinary caching, but it’s far more complex than web caching. Ordinary caching typically only needs to check if a key is a hit; KV Cache must also consider:

1. Is the prefix exactly identical? Token sequences, tokenizer, template, system prompt, and tool descriptions can all affect the cache key.
2. Is the restoration cost lower than the recomputation cost? Restoring KV Cache from GPU, CPU, remote memory, or SSD incurs data transfer overhead.
3. Does the cache crowd out GPU memory? Keeping too much cache can reduce the available batch size for current requests.
4. Does the hit occur on the prefill bottleneck? If the request is mainly slow due to decode, a prefix hit doesn’t necessarily improve overall latency.
5. Does it affect correctness? Reusing KV Cache across different models, LoRAs, tokenizers, or position encodings can lead to errors.

In short: KV Cache reuse is not simply “saving intermediate results.” It’s a system-level problem spanning scheduling, memory management, I/O, routing, template standardization, and online metrics.

Core Principles: From Prefix Caching to Multi-Tier KV Offloading

1. Prefix Caching: Reuse Shared Prefixes, Skip Redundant Prefill

Prefix Caching works on a simple idea: if a new request shares the same prefix as a historical request, directly reuse the already-computed KV Cache, allowing the new request to skip the prefill for the shared portion.

Typical applicable scenarios:

• Fixed system prompt + different user questions
• Same long document + multiple questions
• Multi-turn conversations repeatedly carrying historical context
• RAG requests hitting the same document chunks
• Agent frameworks with stable tool descriptions, role settings, and constraint instructions

vLLM’s Automatic Prefix Caching is a representative implementation. It caches the KV Cache of past requests. When a new request shares the same prefix, it directly reuses the cache, skipping computation for the shared part.

However, the benefit boundary of Prefix Caching is clear: it mainly reduces prefill time, not the decode time for generating new tokens. If user requests don’t share a prefix, or if the output is very long and decode is the main bottleneck, the speedup is diluted.

2. RadixAttention: Managing Complex Reuse Patterns with a Prefix Tree

In real-world business, shared prefixes aren’t always as simple as “the entire system prompt is the same.” Agents, tree search, multi-turn dialogues, few-shot examples, and self-consistency sampling create more complex reuse structures.

SGLang’s RadixAttention uses a radix tree to manage token prefixes and their corresponding KV Cache. This allows the system to not only recognize full prefix hits but also automatically perform prefix matching, cache insertion, and eviction within complex call chains.

Its engineering significance is that the frontend doesn’t need to write extensive caching logic; the runtime can automatically maintain reusable KV Cache based on token prefixes. For programmatic LLM calls, this is more reliable than “each interface building its own cache key.”

3. PagedAttention: Managing KV Cache in Blocks

Another problem with KV Cache is GPU memory fragmentation. Different requests have different context lengths and generation lengths. If a large contiguous block of memory is reserved for each request in advance, a significant amount of space is wasted.

PagedAttention borrows the concept of paging from operating systems. It splits the KV Cache into fixed-size blocks and manages the cache using a mapping from logical blocks to physical blocks. This way, requests don’t need contiguous physical memory, and the system can allocate, reuse, and release KV blocks on demand.

This solves the memory layout and fragmentation problem. It is not the same as Prefix Caching, but the two are often used together: the former makes KV Cache easier to manage, while the latter allows shared prefixes to be reused.

4. KV Offloading: Placing Cache in Larger Tiers

When contexts are very long and concurrency is high, relying solely on GPU memory to store KV Cache is insufficient. Completed KV blocks can be offloaded to CPU memory, file systems, remote memory, or SSDs, and then restored to the GPU upon a hit.

The core trade-offs for these schemes are:

Storage Tier	Advantage	Disadvantage
GPU Memory	Lowest latency, fastest restore	Smallest capacity, highest cost
CPU Memory	Larger capacity, controllable latency	Requires GPU-CPU transfer for restore
SSD/File System	Extremely large capacity, low cost	High I/O latency, random access impacts TTFT
Remote Cache	Can be shared across instances	Network latency, routing & consistency issues

vLLM’s KV Offloading Usage Guide describes the OffloadingConnector as an extension to prefix caching: completed KV blocks can be offloaded to CPU host memory and optionally to secondary storage, and then promoted back to GPU upon a hit. LMCache further positions KV Cache as a shared middle layer across engines and requests, serving prefix reuse and prefill-decode disaggregation.

Engineering Implementation: An Executable KV Cache Reuse Plan

Step 1: Standardize the Prompt Structure First

The first prerequisite for KV Cache reuse is a stable prefix. If upstream requests dynamically concatenate timestamps, random IDs, unordered JSON, or temporary tool descriptions, the cache hit rate will be poor.

It’s recommended to divide the prompt into four segments:

1. Stable System Segment: Role, guidelines, output format, tool descriptions.
2. Stable Knowledge Segment: Long documents, product manuals, codebase summaries, FAQs.
3. Conversation History Segment: Reusable historical parts of multi-turn dialogues.
4. Current Request Segment: The user’s current question and temporary variables.

Only the first three segments are suitable for focused reuse. The fourth segment should generally not be included in a long-term caching strategy.

Example structure:

[stable_system_prompt] [stable_tool_schema] [stable_retrieved_document]
[conversation_history] [current_user_question]

Step 2: Enable Prefix Caching First, Then Monitor Hit Rate

Using vLLM as an example, you can first enable Automatic Prefix Caching in an experimental environment:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,
)

params = SamplingParams(temperature=0.2, max_tokens=512)

prompts = [
    shared_prefix + "\nQuestion: Please summarize Chapter 1.",
    shared_prefix + "\nQuestion: Please list the key terms.",
]

outputs = llm.generate(prompts, params)

Before going live, don’t just look at single response times. Pay close attention to the following metrics:

• prefix cache hit rate
• Average TTFT
• P95 / P99 TTFT
• prefill tokens per second
• GPU KV cache usage rate
• Number of evictions
• Whether a cache hit actually reduces prefill time

Step 3: Route Reusable Content to the Same Set of Instances

If requests are randomly distributed by a load balancer, even if they hit the same prefix, they might land on an instance without the corresponding cache. Production environments need cache-aware routing.

Simple strategies:

• Use consistent hashing based on tenant ID, document ID, knowledge base ID, or session ID.
• Route questions about the same long document to the same set of inference instances as much as possible.
• Pre-warm hot documents on multiple instances.
• Separate high-reuse and low-reuse business into different pools to avoid cache pollution.

This layer is often more easily overlooked than model parameter tuning, but it has a huge impact on hit rate.

Step 4: Evaluate Offloading, Don’t Jump Straight to SSD

Offloading is not a default benefit. It’s suitable for scenarios where GPU memory cannot hold enough KV Cache, but there is a sufficient amount of repeated context.

Recommended order for going live:

1. Enable only in-GPU Prefix Caching, measure hit rate and TTFT.
2. Add a CPU tier, observe if the restore latency is lower than the recomputation cost.
3. Then evaluate FS / SSD / Remote KV Store.
4. Set different max_offload_tokens for different businesses, offloading only high-reuse prefixes.

Example vLLM offloading configuration:

vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "block_size": 64,
      "cpu_bytes_to_use": 1000000000
    }
  }'

The most common mistake here is offloading all tokens. A better practice is to cache only stable prefixes, such as system prompts, tool schemas, and long document bodies, rather than putting the current user question and one-shot intermediate reasoning into long-term cache.

Applicable Scenarios

Scenario	Reuse Pattern	Expected Benefit
Long Document Q&A	Same document + different questions	Significant reduction in prefill time
Multi-turn Dialogue	Reuse conversation history	Noticeable TTFT reduction
Agent & Tool Calling	Stable tool descriptions and policies	Reduced redundant prefill
RAG Hot Knowledge Base	High-frequency document chunks	High hit rate with document ID routing
Few-shot & Evaluation	Fixed examples + varying questions	Significant batch processing benefits

Long Document Q&A

The same PDF, manual, contract, or code documentation is repeatedly queried. Here, the document content is a stable prefix, and the user question is just a tail variation. Prefix Caching benefits are usually significant.

Multi-turn Dialogue

When each request carries the complete conversation history, the historical part keeps growing. If the server can identify and reuse the KV Cache from previous turns, TTFT can be significantly reduced.

Agent & Tool Calling

Agent frameworks often include long tool descriptions, JSON schemas, behavior constraints, and system policies in every request. This content is stable and repetitive, making it suitable for caching.

RAG Hot Knowledge Base

If a large number of user queries hit the same set of knowledge chunks, routing and cache pre-warming by document ID can reduce redundant prefill.

Few-shot & Evaluation Tasks

Few-shot tasks with fixed examples and varying questions are naturally suited for prefix reuse. Evaluation platforms, batch processing tasks, and classification tasks can all benefit.

Common Misconceptions

Misconception	Truth
Prefix Caching speeds up all phases	Only saves prefill for shared prefixes; limited effect on long-output decode
High cache hit rate always saves money	Cache must be restored after a hit; slow I/O can exceed recomputation cost
Same text guarantees KV Cache reuse	Requires consistent tokenizer, template, model version, LoRA, dtype, and block size
More offloading is always better	Consumes CPU memory and I/O; low-reuse content increases latency
Only look at average latency	Easy to improve average but worsen P95/P99; must track tail latency

Misconception 1: Prefix Caching speeds up all phases

No. It mainly saves prefill computation for shared prefixes. If the bottleneck is long-output decode, the effect is limited.

Misconception 2: High cache hit rate always saves money

Not necessarily. The cache must be restored after a hit. If the restore path involves remote storage, slow SSDs, or a network, the cost can exceed recomputation.

Misconception 3: Same text guarantees KV Cache reuse

Not enough. You also need consistent tokenizer, chat template, model version, position encoding strategy, LoRA adapter, parallelism configuration, dtype, and block size.

Misconception 4: More offloading is always better

Offloading consumes CPU memory, I/O bandwidth, and restore scheduling resources. For one-shot requests, low-reuse content, or short prompts, offloading can actually increase latency.

Misconception 5: Only look at average latency

KV Cache-related optimizations easily improve averages but worsen tail latency. You must monitor P95/P99 TTFT, I/O queues, eviction storms, and GPU stalls.

Go-Live Checklist

Prompt & Cache Key

• Is the fixed system prompt completely stable?
• Is the tool schema sorted stably?
• Do RAG documents have stable document IDs and chunk IDs?
• Is the chat template versioned?
• Are the tokenizer, model version, and LoRA adapter included in the cache key?

Routing & Instance Pools

• Is cache-aware routing implemented by session, document, or tenant?
• Is there a pre-warming strategy for hot knowledge?
• Are high-reuse and low-reuse businesses in separate pools?
• Is cache cold start acceptable after instance restart?

Memory & Offloading

• Is the GPU KV cache watermark observable?
• Is the CPU tier capacity larger than the actual reusable prefix size?
• Have the FS/SSD thread counts been stress-tested?
• Is the offload block size compatible with the GPU block size?
• Is there a limit to offload only the first N stable tokens?

Metrics & Alerts

At a minimum, you should have these metrics:

prefix_cache_hit_rate
prefix_cache_hit_tokens
prefix_cache_miss_tokens
ttft_avg / ttft_p95 / ttft_p99
kv_cache_gpu_usage_bytes
kv_cache_cpu_usage_bytes
kv_cache_offload_read_latency
kv_cache_offload_write_latency
kv_cache_eviction_count
gpu_stall_due_to_kv_restore

The alert focus should not be “cache is full,” but rather: after the cache is full, are there frequent evictions? Do evictions cause a drop in hit rate? Does the drop in hit rate push up TTFT?

A Practical Decision Formula

Whether KV Cache reuse is worthwhile can be judged with a simplified formula:

reuse_gain = recompute_prefill_cost - restore_kv_cost - cache_management_cost

Caching is only worth keeping when reuse_gain > 0 and the hit frequency is high enough.

A more engineering-oriented approach is to calculate per business dimension:

worth_cache = shared_prefix_tokens × reuse_count × prefill_cost_per_token
              > restore_bytes / effective_bandwidth + eviction_penalty

This is not an exact model, but it helps teams avoid the misconception of “cache everything possible.”

FAQ

Are Prefix Caching and PagedAttention the same thing?

No. Prefix Caching solves the problem of recomputing repeated prefixes. PagedAttention solves the problem of block-level memory management and fragmentation for the KV Cache. Practical serving frameworks may use both simultaneously.

Should RAG scenarios cache user questions?

Generally, no. User questions vary greatly and have low reuse rates. It’s better to cache stable document chunks, system prompts, tool descriptions, and fixed few-shot examples.

Is KV Offloading suitable for small models?

Not necessarily. For small models, short contexts, and low-reuse requests, the recomputation cost may be lower than the restoration cost. Offloading is more suitable for long contexts, high reuse, and scenarios with insufficient GPU memory.

How do I know if my caching strategy is effective?

Look at four metrics: number of hit tokens, whether TTFT has decreased, whether P99 is stable, and whether GPU memory supports a higher batch size. Looking at hit rate alone is not enough.

Summary

The key to KV Cache reuse is not “store more cache,” but to design stable prefixes, instance routing, memory management, I/O restoration, and business hit rate together.

A solid implementation sequence:

1. First, standardize the prompt to reduce meaningless prefix fluctuations.
2. Enable Prefix Caching and validate it on high-reuse scenarios like long documents, multi-turn dialogues, and Agent tool descriptions.
3. Add cache-aware routing so that the same documents or sessions land on the same set of instances as much as possible.
4. Introduce CPU / FS / SSD Offloading only when GPU cache is insufficient.
5. Use TTFT, P99, hit token count, I/O restore latency, and eviction behavior to determine if you are truly saving money.

For long-context LLM services, KV Cache has evolved from an “internal detail of the inference framework” to a “cost control point at the system architecture level.” Whoever can stably identify, route, save, and restore reusable context will be able to serve more requests within the same GPU budget.

Key References

1. vLLM Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/
2. vLLM KV Offloading Usage Guide: https://docs.vllm.ai/en/latest/features/kv_offloading_usage/
3. LMSYS: Fast and Expressive LLM Inference with RadixAttention and SGLang: https://www.lmsys.org/blog/2024-01-17-sglang/
4. PagedAttention paper: https://arxiv.org/abs/2309.06180
5. LMCache paper: https://arxiv.org/abs/2510.09665
6. KV Cache Offloading for Context-Intensive Tasks: https://arxiv.org/abs/2604.08426
7. Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving: https://arxiv.org/abs/2605.03375
8. Comparative Characterization of KV Cache Management Strategies for LLM Inference: https://arxiv.org/abs/2604.05012