Prefix Caching in Production: A Complete Guide to Reusing KV Cache for RAG and Agent Services

The Problem: The Hidden Cost of Repeated Prefill in LLM Serving

When optimizing LLM inference, most teams first think of faster frameworks, quantization, continuous batching, or splitting prefill and decode phases. However, there’s a more insidious waste in production: a large number of requests repeatedly carry identical prefix content.

These prefixes can be:

• Fixed system prompts and safety rules
• The same long document queried repeatedly in RAG scenarios
• Stable tool lists, function schemas, and workflow instructions in Agent scenarios
• Conversation history resubmitted every turn in multi-turn dialogues
• Stable JSON Schemas or format constraints in structured output scenarios

Transformer inference is typically divided into two phases: prefill and decode. The prefill phase processes the input prompt, computing the Key/Value tensors (the KV Cache) for each attention layer. The longer the input, the heavier the prefill. RAG, Agent, and long-document QA systems pack a lot of context into the prompt. Processing the exact same prefix from scratch for every request wastes GPU compute and increases TTFT (Time To First Token).

Prefix Caching has a straightforward goal: when a new request shares a common prompt prefix with a historical request, directly reuse the already-computed KV Cache, skipping the prefill computation for the shared part.

This is not “response caching.” A response cache hit returns the old answer directly. With a Prefix Caching hit, the model still reads the subsequent dynamic input and generates a new answer. It reuses intermediate computation results, not the final text.

---

Core Principle: Caching Prefix KV, Not Answers

1. What is a Reusable Prefix?

In an autoregressive Transformer, each token in the prefill phase generates Key and Value tensors for its corresponding layer. If two requests have their first N tokens exactly the same, the KV tensors for those N tokens can be reused.

A typical RAG request can be split into two parts:

[Fixed System Instruction + Tool Definitions + Document Context] + [Current User Question]

As long as the left part is token-identical, subsequent requests have a chance to hit the cache. vLLM’s Automatic Prefix Caching (APC) documentation explicitly states: APC caches the KV Cache of existing queries. When a new query shares a common prefix with an existing query, it can reuse this cached KV Cache, skipping the computation for the shared part.

2. Why “Token-Level Identity” is Crucial

Prefix Caching has no tolerance for text that “looks similar.” An extra space, a different field order, a timestamp placed at the beginning, or a changed tool schema order can all lead to a different token sequence, causing a cache miss.

Therefore, the key to engineering success is not simply flipping a parameter, but restructuring how you organize your prompts:

Static Content (High Reuse)	Dynamic Content (Low Reuse)
System Prompt	User Question
Tool Definitions & Schema	Current Time
Output Format Constraints	Session Variables
Long Document Context	Personalization Parameters
Few-shot Examples	Trace ID

Principle: Place static content stably at the beginning of the prompt, and dynamic content as late as possible. The OpenAI Prompt Caching documentation also recommends placing static instructions and examples at the start and user-specific information at the end, noting that cache hits depend on an exact prefix match.

3. Prefix Caching Optimizes Prefill, Not Decode

This point is often misunderstood. The vLLM documentation clearly states that APC only reduces the time for query processing (the prefill phase), not the decode time for generating new tokens.

Therefore, Prefix Caching provides the most benefit in these scenarios:

• Long input, short output
• The same long document is queried with different questions
• Multi-turn sessions with long history appended each turn
• Agent tool definitions are large, but only a few task parameters change per turn

For scenarios with very long outputs and non-repeating input prefixes, the benefits are limited.

---

Engineering Implementation: From “Enabling Cache” to “Designing Cacheable Prompts”

1. vLLM Scenario: Enable First, Then Observe

Enabling Prefix Caching in vLLM is simple: set enable_prefix_caching=True.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_prefix_caching=True,
)

sampling_params = SamplingParams(temperature=0.2, max_tokens=512)

shared_context = """You are a production support assistant.
Follow the company policy below...
[long static document or tool policy]"""

prompts = [
    shared_context + "\nUser question: How should I handle case A?",
    shared_context + "\nUser question: What about case B?",
]

outputs = llm.generate(prompts, sampling_params)

This is just a minimal example. In production, you also need to monitor:

• Number of cache hit tokens
• Prefill latency / TTFT
• Decode latency / TPOT
• GPU KV Cache usage
• Cache eviction count
• Hit rate for different business units, tenants, models, and prompt templates

If you only look at overall QPS, you might mistakenly attribute the effects of continuous batching, model quantization, or changes in request length to Prefix Caching.

2. OpenAI API Scenario: Stabilize the Prefix and Read cached_tokens

OpenAI Prompt Caching is enabled automatically, but not all prompts will benefit. The official documentation states that caching is available for prompts of ≥1024 tokens, and the usage field will return cached_tokens.

It’s recommended to log the following fields:

{
  "model": "gpt-5.5",
  "prompt_tokens": 2006,
  "cached_tokens": 1920,
  "ttft_ms": 430,
  "total_latency_ms": 1850,
  "prompt_template_version": "support-agent-v7"
}

The key metric for the business side isn’t “is caching on?”, but:

cached_tokens / prompt_tokens

The higher this ratio, the more effectively repeated prefill is being reused. If the ratio is consistently near zero, the problem is usually not the model, but the prompt structure or routing strategy preventing a stable prefix hit.

3. SGLang / RadixAttention: Organize Common Prefixes into a Tree

The SGLang paper introduces RadixAttention for reusing KV Cache across multiple generation calls. Its core idea is to organize shared prefixes using a structure similar to a radix tree, allowing multiple requests with common token prefixes to share already-computed intermediate states.

This direction is particularly valuable for complex LLM programs, as Agents, RAG, multi-branch reasoning, few-shot selection, and structured output often involve not a single call, but a set of related calls. If the runtime can identify shared prefixes between these calls, it can significantly reduce redundant computation.

4. Version Your Prompt Templates

The stability of Prefix Caching is highly dependent on your prompt template. If you adjust system prompts, tool schema ordering, field names, or example order with every release, your cache hit rate will drop suddenly.

Treat prompt templates as observable, rollback-able engineering objects:

prompt_template:
  id: support-agent-v7
  model: llama-3.1-8b-instruct
  static_prefix_hash: sha256:...
  tool_schema_version: tools-2026-06-30
  output_schema_version: answer-json-v3
  cache_strategy: prefix-cache-first

After deployment, observe TTFT and cached token ratio by template version to avoid the problem of “changing a prompt and suddenly seeing higher latency.”

---

Applicable Scenarios: Which Use Cases Benefit Most

Scenario	Benefit Mechanism	Typical Metric Improvement
Long Document QA	Contracts/manuals/policies as a fixed, reusable prefix	TTFT reduction of 50-90%
Multi-turn Dialogue	Stable history prefix reused in subsequent turns	Significant latency reduction for 2nd+ turns
Agent Tool Calling	Highly repetitive tool schemas, permission descriptions, call constraints	Prefill overhead approaches 0
Structured Output	Fixed prefixes like JSON Schema, classification labels	Reduced latency for structured output

Long Document QA

When the same contract, manual, policy, or technical document is queried repeatedly, the long document itself becomes a highly reusable prefix. The vLLM documentation lists long document query as a typical benefit scenario for APC: the same long document only needs to be processed once, and future requests reuse its KV Cache.

Multi-turn Dialogue

In multi-turn conversations, a new request usually includes the previous chat history. If the history prefix remains stable, subsequent turns can reuse the already-processed content. Note that summarizing, truncating, or reordering the history each turn can break prefix consistency.

Agent Tool Calling

Agent tool definitions, permission descriptions, call constraints, and output formats are often long, but the same Agent template is highly similar across multiple requests. Placing the tool schema stably at the prefix position can significantly increase the cache hit probability.

Structured Output

JSON Schemas, classification label descriptions, and field constraints are often fixed and reusable. The OpenAI documentation also lists structured outputs schemas as cacheable content, as the schema participates in caching as part of the system message prefix.

---

Common Misconceptions

Misconception 1: Treating Prefix Caching as Semantic Caching

Prefix Caching does not judge whether two questions are semantically similar; it only checks if the prefix tokens match. If you want to cache “answers to similar questions,” that’s the domain of semantic cache or response cache, which have completely different risks and governance models.

Misconception 2: Placing Dynamic Fields at the Start of the Prompt

Many systems place the current time, user ID, trace ID, experiment group, or random nonce at the beginning of the prompt. This directly destroys prefix reuse. Dynamic fields should be placed as late as possible, and log fields should not be in the model prompt at all.

Misconception 3: Only Flipping the Parameter, Not Changing Prompt Structure

Enabling enable_prefix_caching is just the first step. The real effectiveness is determined by prompt structure, routing strategy, template stability, and cache capacity management.

Misconception 4: Ignoring Cache Eviction Strategy

GPU KV Cache space is limited, and cache blocks will be evicted. The SAECache paper points out that the reuse rate of different token types varies greatly—system prompts, user queries, tool outputs, and model replies have different cache values. A production system should at least monitor by business type, template version, and cache hit rate to prevent high-value prefixes from being evicted by low-value, long-tail requests.

---

Production Deployment Checklist

Prompt Structure

• Are static system instructions placed at the very beginning?
• Is the tool schema sorted stably?
• Are few-shot examples stable?
• Are current time, user variables, and trace IDs placed at the end or removed from the prompt?
• Is the RAG document concatenation order deterministic?
• Is the template version rollback-able?

Inference Service

• Is prefix caching enabled?
• Is the number of cached tokens being logged?
• Are prefill latency and decode latency distinguished?
• Are TTFT, TPOT, P95, and P99 being monitored?
• Are KV Cache hit rate and eviction rate being observed?
• Are metrics broken down by model, tenant, business line, and template version?

Business Governance

• Is sharing sensitive context across tenants avoided?
• Is the caching strategy confirmed to comply with data retention requirements?
• Is there a canary process for template releases?
• Are prompt templates rolled back in sync with model rollbacks?
• Are optimization tasks created for templates with low hit rates?

---

Reference Architecture: Making Prefix Caching a Platform Capability

A robust implementation involves adding two layers—Prompt Assembly and Cache Metrics—on top of your LLM Gateway or Inference Router:

Business Request
  → Prompt Assembly: Fixed Prefix + Dynamic Suffix
    → Template Versioning: Record template version and prefix hash
      → Inference Router: Route to inference backend supporting prefix caching
        → LLM Engine: vLLM / SGLang / API Provider
          → Metrics: cached_tokens, TTFT, TPOT, eviction, hit ratio

This has two main benefits:

1. Business teams don’t need to understand the underlying KV Cache mechanism.
2. The platform team can use unified metrics to determine which prompt templates are worth optimizing.

---

FAQ

Does Prefix Caching change the model’s output?

Normally, no. It reuses the intermediate KV representation of the same prefix; the subsequent dynamic input still participates in the full inference. The OpenAI documentation also states that Prompt Caching does not affect output token generation or the final response; the output should be consistent regardless of a cache hit.

What is the relationship between Prefix Caching and KV Cache Quantization?

They solve different problems. Prefix Caching focuses on reusing the KV of the same prefix to reduce repeated prefill. KV Cache Quantization focuses on compressing the KV Cache footprint to reduce memory pressure. They can be used together, but you should monitor hit rate, precision, latency, and memory separately when deploying them.

Why is my cache hit rate so low?

Common reasons include: dynamic fields at the start of the prompt, unstable tool schema ordering, non-deterministic RAG document sorting, frequent template changes, context truncation breaking the history prefix, and different tenants or requests being routed to different cache instances.

Is Prefix Caching suitable for all Agents?

No. If an Agent dynamically generates its tool list, dynamically rewrites its system prompt, or has vastly different context each turn, the hit rate will be very low. It is best suited for production Agents with a stable tool set, fixed workflow, and relatively long context.

---

Key References

1. vLLM - Automatic Prefix Caching — docs.vllm.ai
2. OpenAI API Docs - Prompt Caching — developers.openai.com
3. SGLang: Efficient Execution of Structured Language Model Programs — arXiv:2312.07104
4. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — arXiv:2402.15220
5. Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches — arXiv:2605.18825
6. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — arXiv:2510.09665
7. LLM Query Scheduling with Prefix Reuse and Latency Constraints — arXiv:2502.04677