LLM Tool Result Caching in Production: Reducing Agent External Call Costs with Idempotency Keys, TTL, and Invalidation Strategies

Background: Agent Costs Aren’t Just Model Tokens

When optimizing LLM applications, many teams first think of compressing prompts, switching to smaller models, using Batch APIs, or reducing context length. These methods are valuable, but for agent-based applications, another cost is often underestimated: external tool calls.

A customer service agent might repeatedly query orders, membership benefits, or logistics tracking. A data analysis agent might call SQL, search APIs, or internal configuration services multiple times. A research agent might perform repeated web searches for similar keywords. The model itself just “decides which tool to call”; the real time, money, rate limiting, and instability often come from the external systems behind the tools.

OpenAI’s Function Calling documentation describes tool calling as a multi-step process where the model interacts with external systems: first, callable tools are provided to the model, the model returns a tool call, the application executes the code, and then the tool result is fed back to the model. Anthropic’s tool use documentation also emphasizes that tools can be executed on the application side or by server-side tools, and server-side tools may incur additional usage costs. Therefore, tool result caching is not an edge optimization; it’s a foundational capability that agent platforms must consider when going into production.

This article discusses Tool Result Cache, which caches the result of a specific tool call given a particular input and context boundary. It differs from Prompt Cache, KV Cache, and LoRA adapter cache. It sits between the Agent Runtime and external tools, aiming to reduce duplicate calls, lower tail latency, mitigate third-party API rate limits, and keep results auditable, invalidatable, and replayable.

Core Principle: Cache Verifiable Tool Outputs, Not Natural Language

Tool Call Results Must Be Structured

Do not cache the natural language answer generated by the model as the tool result. Production systems should cache the raw structured data returned by the tool, for example:

{
  "tool_name": "get_order_status",
  "arguments": {
    "tenant_id": "t_001",
    "order_id": "O20260707001"
  },
  "result": {
    "status": "shipped",
    "carrier": "SF Express",
    "updated_at": "2026-07-07T02:40:00-04:00"
  },
  "source": "order-service-v3",
  "fetched_at": "2026-07-07T03:00:00-04:00"
}

The model can rephrase the language each time; the factual data returned by the tool is what should be cached, verified, and tracked.

Cache Keys Must Consist of “Tool Name + Normalized Parameters + Isolation Boundary”

The most common tool caching accident is not that cache invalidation is too slow, but that the cache key design is too coarse. A seemingly identical query may yield different results across tenants, users, permissions, languages, time windows, or tool versions.

A recommended cache key should include at least the following:

tool_result:v1:{tenant_id}:{user_scope}:{tool_name}:{tool_version}:{normalized_args_hash}:{policy_scope}

Here, normalized_args_hash should come from sorted JSON parameters, not the raw string generated by the model. For example, {"city":"Paris","unit":"celsius"} and {"unit":"celsius","city":"Paris"} should hit the same key.

import hashlib, json

VOLATILE_FIELDS = {"trace_id", "request_id", "timestamp"}

def normalize_args(args: dict) -> str:
    stable = {k: v for k, v in args.items() if k not in VOLATILE_FIELDS}
    return json.dumps(stable, ensure_ascii=False, sort_keys=True, separators=(",", ":"))

def build_cache_key(tenant_id: str, user_scope: str, tool_name: str,
                    tool_version: str, args: dict) -> str:
    digest = hashlib.sha256(normalize_args(args).encode("utf-8")).hexdigest()[:32]
    return f"tool_result:v1:{tenant_id}:{user_scope}:{tool_name}:{tool_version}:{digest}"

The user_scope is critical. For public weather queries, use public; for permission-related tools like orders, accounts, or reports, must bind to the user, role, or authorization scope, otherwise cross-user cache leaks can occur.

TTL Is Not a Uniform Configuration; It’s Part of the Tool’s Capability

Caching systems like Redis support writing strings with expiration times, but agent tool results cannot simply be set to “expire in 10 minutes.” Each tool should declare its own freshness contract.

Tools can be layered by timeliness:

tools:
  get_product_detail:
    cacheable: true
    ttl_seconds: 3600
    invalidation: product_updated_event
  get_weather:
    cacheable: true
    ttl_seconds: 600
    invalidation: ttl_only
  get_order_status:
    cacheable: true
    ttl_seconds: 30
    invalidation: order_status_changed_event
  send_refund:
    cacheable: false
    reason: side_effect

TTL should come from business semantics, not the caching system’s default. Product details can be cached longer, order status may only allow a short TTL, and tools with side effects like payments, refunds, emails, or order placement should not have their execution results cached, nor should cache hits replace actual execution.

Engineering Implementation: Place an Auditable Cache Layer Before the Tool Runtime

Recommended Pipeline

The production pipeline can be broken down into five steps:

1. The model returns a tool call.
2. The Tool Runtime performs permission checks, parameter validation, and idempotency checks.
3. The Cache Policy determines whether to allow reading from the cache.
4. If a hit, return the cached result; if a miss, call the real tool.
5. Regardless of hit or miss, feed the result back to the model as a standard tool output and write to the trace.

The key point is: the cache layer should not make the model aware that “the tool was not executed.” To the model, it still receives a tool output; on the engineering side, the trace must explicitly mark cache_hit=true, cache_key, ttl_remaining, source_fetched_at, and tool_version.

async def execute_tool_with_cache(tool_call, context):
    policy = registry.get_policy(tool_call.name)
    if not policy.cacheable:
        result = await real_tool_execute(tool_call, context)
        return ToolOutput(result=result, cache_hit=False)

    cache_key = build_cache_key(
        tenant_id=context.tenant_id,
        user_scope=context.user_scope,
        tool_name=tool_call.name,
        tool_version=policy.version,
        args=tool_call.arguments,
    )
    cached = await cache.get(cache_key)
    if cached and not policy.must_revalidate(cached, context):
        return ToolOutput(result=cached["result"], cache_hit=True, cache_key=cache_key)

    result = await real_tool_execute(tool_call, context)
    await cache.set(cache_key, {
        "result": result,
        "tool_version": policy.version,
        "fetched_at": now_iso(),
        "scope": context.user_scope,
    }, ttl=policy.ttl_seconds)
    return ToolOutput(result=result, cache_hit=False, cache_key=cache_key)

Idempotency Keys and Cache Keys Should Not Be Confused

Idempotency keys are used to prevent duplicate execution of the same side-effect request, such as duplicate refund submissions, duplicate SMS sends, or duplicate ticket creation. Cache keys are used to reuse read-only results. Both can share normalized parameter logic, but their semantics are entirely different.

A common mistake is: seeing the model call create_ticket again, and directly returning the previous result. The correct approach is: if the business request carries the same idempotency key, return the same transaction result; if there is no idempotency key, write operations should not be treated as cache hits.

A simple rule of thumb: if tool execution changes the external world, do not use normal result caching.

Invalidation Strategies Should Support Event-Driven Approaches

Relying solely on TTL leaves a significant window. For example, an order status changes from “pending shipment” to “shipped,” but the cache still has 20 seconds before expiration. This might be acceptable for general Q&A, but not for scenarios like customer service, risk control, or payments.

A more robust approach is to combine three types of strategies:

Strategy	Use Case	Advantages	Risks
Short TTL	Frequently changing but low-risk data	Simple to implement, no extra dependencies	Window for serving stale data
Event-Driven Invalidation	Critical business state changes	Good real-time performance, precise invalidation	Depends on event infrastructure reliability
Version Prefix Invalidation	Tool upgrades, metric changes	Batch invalidation, avoids dirty reads	Requires version management discipline

Don’t aim for a “perfect invalidation” design upfront. The more important thing is to have each tool declare its cache level, TTL, invalidation events, and risk level.

Applicable Scenarios

Tool result caching is suitable for these scenarios:

• Repeated queries for the same order, account, configuration, or knowledge base snippet across multiple conversation turns.
• Agents repeatedly calling the same search, SQL, or internal API during the planning phase.
• Third-party APIs with strict QPS, billing, or stability limits.
• Multiple parallel agent tasks accessing the same read-only data.
• Evaluation, replay, or canary testing that needs to reuse stable tool outputs.

Recent research like ToolCaching also frames the problem directly: LLM tool-calling involves duplicate or redundant calls, but traditional caching strategies are hard to adapt because tool requests are semantically heterogeneous, workloads are dynamic, and result freshness requirements vary. TVCACHE further points out that in post-training or rollout scenarios for agents, tool outputs depend on prior interaction states, so the full tool history must be considered, not just the current parameters.

The takeaway for production systems is: don’t make tool caching a generic Redis wrapper. It should be a first-class capability of the Agent Runtime, capable of understanding tool semantics, state boundaries, and business risks.

Common Pitfalls

Pitfall 1: Caching All GET-Type Tools by Default

The HTTP method is not a sufficient condition. A GET endpoint may depend on user permissions, real-time status, geographic location, A/B experiment groups, or backend configuration. Before caching, you must answer three questions: Is the result read-only? Is it reusable? Does it have a clear time boundary?

Pitfall 2: Only Tracking Hit Rate, Ignoring Error Reuse Rate

A high cache hit rate is not necessarily good. If the cache is serving stale, unauthorized, or erroneous data, the higher the hit rate, the faster the incident spreads. Production metrics should at least include:

Metric	Description
cache_hit_rate	Cache hit rate
cache_stale_hit_count	Number of times stale data was served
cache_bypass_count	Number of times cache was bypassed
cache_eviction_count	Number of cache evictions
tool_latency_saved_ms	Latency saved from tool calls
external_api_calls_saved	Number of external API calls saved
cache_error_reuse_count	Number of times erroneous results were reused
per_tool_cache_hit_rate	Cache hit rate per tool

Pitfall 3: Not Writing Traces on Cache Hits

A cache hit is not “nothing happened.” It changes the agent’s execution path and affects the final answer. Every hit should be recorded in the trace, so that during replay, you know whether the model saw a real-time tool result or a historical cached result.

Pitfall 4: Storing Tool Outputs As-Is for Long Periods

Tool returns may contain PII, permission data, or commercially sensitive information. The cache layer must inherit data classification policies: mask what needs masking, encrypt what needs encryption, use short TTLs where appropriate, and strictly isolate data that should not be reused across tenants.

Pre-Launch Checklist

Before going live, verify each item on this checklist:

• Every tool has cacheable, ttl_seconds, risk_level, and tool_version configured.
• Write operations and side-effect operations have result caching disabled by default.
• Cache keys include tenant, user authorization scope, tool name, tool version, and a digest of normalized parameters.
• Parameter normalization removes volatile fields like trace_id, request_id, and timestamp.
• Permission changes, data updates, and tool version upgrades can trigger invalidation or version prefix switching.
• Traces record cache_hit, cache_key_hash, ttl_remaining, and source_fetched_at.
• During canary testing, observe hit rate, latency savings, error reuse, and external API call reduction per tool.
• Support quick cache disabling by tenant, tool, or cache_key_hash.
• Support replaying the same conversation with the option to “use cached results” or “re-execute tools.”

Summary

Tool result caching is an unavoidable engineering foundation for agent platforms moving to production. It’s not a simple Redis SETEX; it’s a strategy system that must deeply integrate with business semantics, permission boundaries, tool versions, and audit trails. The core principles can be summarized in three points:

1. Cache structured tool outputs, not model-generated natural language.
2. Cache keys must include isolation boundaries (tenant, user, tool version), not just parameter digests.
3. Invalidation strategies must be multi-pronged: TTL + event-driven + version prefix, complementary rather than mutually exclusive.

By doing this well, you can significantly reduce external API call costs and latency while keeping agent execution paths auditable, replayable, and governable—without sacrificing correctness.

References

• OpenAI Function Calling Documentation
• Anthropic Tool Use with Claude
• Redis Strings Documentation
• ToolCaching: Towards Efficient Caching for LLM Tool-calling
• TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents