What's the difference between Prompt Caching and regular response caching?

Prompt Caching reuses the model's precomputed results for the same input prefix, typically reducing input token costs and time-to-first-token (TTFT). Response caching reuses complete model answers, suitable for repeated questions.

Why do agent systems tend to waste Prompt Cache more easily?

Each agent turn appends tool calls, observations, timestamps, and temporary state. If these dynamic elements are inserted into the prefix, they break exact prefix matching, invalidating the entire cache.

What should be cached first in production?

Prioritize caching stable system prompts, tool definitions, long document backgrounds, and a few fixed examples. Place dynamic content like user input, tool results, and timestamps after the cache boundary. Monitor cached tokens, cache hit ratio, and TTFT after deployment.

Why is my cached token count still 0 despite using a long prompt?

Common causes: request length below provider threshold, inconsistent prefixes, dynamic fields at the start, request intervals exceeding cache TTL, routing not hitting the same cache location, or the model/API not supporting the caching feature.

Prompt Caching in Practice: Cost and Latency Governance for Long-Context Agents

Background: Long Context Is Not a Free Lunch

Long context has evolved LLM applications from “single-turn Q&A” to “agents with data, tools, and memory.” A real-world business agent often packs system prompts, tool descriptions, permission boundaries, business terminology, conversation summaries, retrieval results, and user questions into a single context. This improves usability but introduces two direct problems: high input token costs and increased TTFT (Time To First Token).

Many teams first think of compressing prompts, reducing tool descriptions, or trimming context. These all work, but there’s one often underestimated optimization: Prompt Caching. It doesn’t cache the final answer; instead, it lets the model server reuse precomputed results for the same prompt prefix. For multi-turn agents, long-document Q&A, fixed toolset calls, and code repository analysis, it can reduce the cost and latency of repeated inputs without altering model output semantics.

However, Prompt Caching isn’t a flip-the-switch optimization. In production, three counterintuitive phenomena often appear:

The provider claims automatic caching, but cached_tokens in the bill is minimal;
Local tests show benefits, but the hit rate fluctuates wildly in production;
Full-context caching seems simplest but can actually increase latency.

The issue isn’t “whether caching exists,” but whether the prompt prefix is stable, the cache boundary is correct, and dynamic content pollutes the reusable part.

Core Principle: Prompt Caching Reuses Stable Prefixes

From an engineering perspective, the core of Prompt Caching is prefix reuse. When processing input, the model performs forward computation on prompt tokens and generates reusable intermediate states. Providers or inference frameworks can skip part of the repeated computation when subsequent requests encounter the same prefix, reducing prefill costs.

1. Exact Prefix Matching

Most Prompt Caching relies on “prefix consistency.” The key to cache hits isn’t “semantic similarity,” but that the beginning of the request matches at the token level. Inserting any changing field—timestamps, UUIDs, temporary tool results, user names, randomly ordered JSON fields—into the prefix can break the hit.

Therefore, the first engineering principle of Prompt Caching is: Static content first, dynamic content last.

Recommended order:

Stable system instructions → Stable security boundaries → Stable tool definitions → Stable business terminology
→ Stable examples → Reusable long documents/knowledge base
--- Cache boundary ---
→ Current user input → Retrieval results → Tool call results → Temporary state → Output format supplements

Not recommended:

Current time: 2026-06-29 10:01:19 | User ID: xxx | Current request ID: uuid
→ Stable system instructions → Stable tool definitions → Long document background → User question

The second approach places high-frequency changing fields at the very front, preventing the stable content behind from forming a stable prefix.

2. Automatic Caching vs. Explicit Cache Boundaries

Caching capabilities vary significantly across providers:

Provider	Caching Method	Characteristics
OpenAI	Automatic caching	Enabled automatically for recent models; once a request reaches a certain length, prefix matching yields caching benefits; emphasizes exact prefix match
Anthropic Claude	Automatic + explicit `cache_control`	Supports automatic caching and explicit cache breakpoints; suitable for multi-turn conversations, long backgrounds, and multi-example tasks
Google Gemini	Implicit + explicit context caching	Implicit caching enabled by default; also provides a context caching API to create reusable cache objects

Conclusion: Don’t treat Prompt Caching as a single feature of one provider. A safer approach is to define a cache-friendly context structure at the application layer, then adapt to automatic caching, explicit cache boundaries, or context cache objects based on the model provider.

3. Prompt Caching Is Not Semantic Cache

Prompt Caching is often confused with common caching layers. The table below clarifies the differences:

Type	Cached Object	Hit Condition	Use Case	Risk
Prompt Caching	Precomputed results for the same prefix	Token-level prefix match	Long system prompts, tool definitions, long documents, multi-turn agents	Prefix broken by dynamic content
Response Cache	Complete model answer	Exact or normalized request match	FAQ, fixed queries, low-variation Q&A	Stale answers, permission leaks
Semantic Cache	Results for semantically similar questions	Vector similarity or semantic judgment	Customer service, knowledge base Q&A	Wrong reuse, semantic drift
Retrieval Cache	Retrieval results	Exact or rewritten query match	RAG, multi-turn search	Document update lag

The value of Prompt Caching is: even if each user question is different, as long as the preceding system prompts, tool definitions, and long background are consistent, a large portion of input processing costs can be reused.

Engineering Implementation: Split Context into “Stable Zone” and “Dynamic Zone”

1. Establish Context Layering

An agent prompt suitable for Prompt Caching can be split into five layers:

Layer 1: Kernel Prompt. Includes role, boundaries, security rules, and output quality requirements, typically unchanged for long periods.

Layer 2: Tool Schema. Includes tool list, parameter schema, call constraints, and error handling rules. Tools change infrequently, but if tool order or descriptions vary dynamically, cache hits are affected.

Layer 3: Domain Pack. Includes business terminology, product rules, fixed knowledge, code repository descriptions, and API specifications. It may be loaded based on task type but should remain stable within the same task category.

Layer 4: Session Summary. A compressed summary of conversation history. More dynamic than previous layers; careful consideration is needed whether to include it in the cache—if the summary is rewritten each turn, it will break subsequent matches.

Layer 5: Turn Context. Current user question, retrieval results, tool results, error stack, current time, request ID, and temporary variables. Should be placed last and is typically not a caching focus.

Example structure:

<stable_prefix>
  <kernel_prompt>
    You are an enterprise task execution agent.
    You must adhere to permissions, auditing, output format, and error handling requirements.
  </kernel_prompt>
  <tool_definitions>
    Tool list with fixed ordering.
    Parameter schema with fixed format.
    Do not insert current turn's tool results here.
  </tool_definitions>
  <domain_pack>
    Fixed business rules.
    Fixed glossary.
    Fixed examples.
  </domain_pack>
</stable_prefix>

<dynamic_context>
  <user_request>Current user input.</user_request>
  <retrieval_results>Current retrieval results.</retrieval_results>
  <tool_observations>Current tool call results.</tool_observations>
</dynamic_context>

2. Avoid Tool Results Polluting the Cache Prefix

The most common place where agent systems break Prompt Cache is tool calls. Tool return results are typically highly dynamic: web search results change, database results change, execution logs change, and timestamps change. If these are inserted before system prompts or tool definitions, the cache is essentially invalidated.

The correct approach is to separate tool definitions from tool results:

# Stable
You can use the following tools:
- search_web(query)
- read_file(path)
- run_test(command)

# Dynamic
Current tool execution results:
...

Also, avoid dynamically reordering the tool list each turn. Even if the tool semantics don’t change, variations in field order, JSON formatting, or description templates can cause cache misses.

3. Don’t Place “Seemingly Harmless” Dynamic Fields in the Prefix

Many low cache hit rate issues stem from engineering details. For example:

{
  "request_id": "a8f9...",
  "timestamp": "2026-06-29T10:01:19-04:00",
  "user": "u123",
  "instructions": "..."
}

In this structure, request_id and timestamp are placed first, preventing the instructions behind from forming a stable prefix. A better approach is:

{
  "instructions": "...",
  "tools": [...],
  "domain_context": "...",
  "runtime": {
    "request_id": "a8f9...",
    "timestamp": "2026-06-29T10:01:19-04:00",
    "user": "u123"
  }
}

For further optimization, place runtime at the very end of the complete prompt, not just after within the JSON.

4. Make Template Rendering Reproducible

Prompt Caching requires stable input, so template rendering must be reproducible. Before deployment, check:

Whether JSON field order is stable;
Whether tool list order is stable;
Whether example order is stable;
Whether multilingual text switches automatically;
Whether there are random spaces, random line breaks, or random separators;
Whether “session summaries” are regenerated each turn;
Whether the current date is written into the system prompt;
Whether user variables are inserted into the beginning of a fixed template.

These issues don’t affect functional correctness but significantly impact caching benefits.

Applicable Scenarios: Where Prompt Caching Is Most Worthwhile

Scenario 1: Long System Prompt Agents

If an agent’s system prompt contains extensive rules, tool descriptions, error handling strategies, and output formats, Prompt Caching often yields significant benefits. Especially when the same agent handles multiple requests in a short period, the stable prefix repeats.

Typical scenarios: coding agents, data analysis agents, deep research agents, ticket handling agents, compliance review agents.

Scenario 2: Multi-Turn Q&A on Fixed Documents

When users ask consecutive questions about the same long document, place the document content or summary in the stable zone and each question after. If the provider supports explicit context caching, the long document can also be created as a cache object.

Suitable for: contract review, paper reading, code repository analysis, product manual Q&A, policy document interpretation.

Scenario 3: Few Fixed Tools, Many Similar Tasks

Many enterprise applications have stable tool sets—e.g., query customer, query order, create ticket, update status. Tool schemas are long, but each turn only varies user input and tool results. This scenario is ideal for placing tool definitions in the cache prefix.

Scenario 4: Batch Processing of Similar Tasks

If you’re processing many similar documents or data—e.g., batch extraction, batch classification, batch SEO article generation—fixed instructions and output schemas can be cached, with individual samples placed after.

Common Misconceptions

Misconception 1: Automatic Caching by Provider Means No Need to Manage Prompt Structure

Automatic caching only works when there’s a stable prefix. If the application layer places dynamic fields at the beginning of the prompt each time, automatic caching cannot achieve stable hits.

Misconception 2: More Caching Is Always Better

In long-context agents, full-context caching isn’t always optimal. Tool results, search results, and observation logs change frequently; caching them may cause many cache writes but few cache reads. Excluding dynamic tool results and controlling cache boundaries is usually more stable than naive full-context caching.

Misconception 3: Prompt Caching Can Replace Context Compression

Prompt Caching reduces the processing cost of repeated prefixes but doesn’t solve the problem of unbounded context growth. Conversation history, tool results, and retrieval snippets still need trimming, summarization, and hierarchical management.

Misconception 4: Only Look at Total Tokens, Not Cached Tokens

After deployment, you must observe cache hits. Looking only at prompt_tokens and total cost makes it hard to determine if optimization is effective. Key metrics to track:

Input token count
Cached token count
Cache hit ratio
TTFT (P50/P95/P99)
Total latency
Ratio of cache writes to cache reads
Hit rate by task type

Misconception 5: Ignoring Security Boundaries

Prompt Caching involves server-side reuse of request content. In production, verify the provider’s isolation policies, organizational boundaries, data retention policies, ZDR support, and whether the gateway layer might introduce shared credential isolation issues. For highly sensitive business, Prompt Caching should be included in security reviews, not just as a cost optimization item.

Production Checklist

Prompt Structure

Are stable system prompts fixed at the very beginning?
Are tool definitions in fixed order?
Are examples in fixed order?
Are all dynamic variables placed after the cache boundary?
Are retrieval results and tool results kept out of the cache prefix?
Are long documents organized into reusable blocks by task type?

Template Stability

Is JSON serialization stable?
Are random UUIDs, timestamps, or trace IDs appearing in the prefix?
Are A/B prompt experiments causing frequent prefix changes?
Is the system prompt rewritten on every request?
Does automatic summarization pollute the cache zone?

Observability Metrics

Security and Compliance

Is caching isolated by organization, project, or subscription?
Is ZDR or equivalent data retention policy supported?
Could the gateway service share organizational credentials?
Is caching of highly sensitive content prohibited?
Are necessary but non-excessive cache observation fields logged?

Conclusion: Caching Benefits Come from Structure, Not a Switch

The key to Prompt Caching isn’t “whether the provider has this feature,” but whether the application can consistently generate stable, reusable, and clearly bounded prompt prefixes. For long-context agents, the most effective strategy is usually not to cache the entire context, but to place system prompts, tool definitions, business rules, and fixed materials at the front, and user input, tool results, retrieval snippets, timestamps, and temporary state at the back.

In production, let metrics speak: watch cached_token, TTFT, cache hit ratio, cache write/read ratio, and task type breakdown. If the cache hit rate is low, don’t switch models first—check if the prompt template is stable.

Prompt Caching is neither magic nor a panacea. It’s part of context engineering. When the structure is correct, it can significantly reduce the recurring costs of long-context applications; when the structure is wrong, it’s just a feature name on the bill with no visible benefit.