Tool Result Caching in Production: Reducing Agent External Call Costs with Cache Hit Rates

Background: When Agents Slow Down, It’s Often Not the Model—It’s the Tool Calls

In production, an LLM Agent’s response is rarely a single model generation. It’s a chain involving the model, tools, external APIs, databases, retrieval systems, and secondary summarization. OpenAI’s Function Calling documentation breaks tool calls into multiple steps: the model may return a tool call after receiving a request, the application executes the function, and then the tool output is sent back to the model for the final response. Anthropic’s Tool Use documentation similarly emphasizes that client-side tools are executed by the application, with the model returning tool_use and the application submitting tool_result.

This means an Agent’s latency and cost come from more than just model tokens. Weather, exchange rates, inventory, order status, customer profiles, permission queries, search results, internal knowledge base retrieval, price calculations, risk rule hits—each external tool call introduces network latency, downstream rate limits, third-party fees, and failure probability. Moreover, many requests are highly repetitive in short periods: the same user repeatedly checking the same order status, the same team frequently querying the same service price, the same customer service scenario repeatedly reading the same policy explanation.

Tool Result Caching addresses this: when the model initiates a tool call that is cacheable, reusable, and has a definable freshness, instead of hitting the external system directly, it first checks the tool result cache. If there’s a hit and it hasn’t expired, the cached result is returned to the model as tool_result. If there’s a miss or it’s expired, the real tool call is executed and the result is written to the cache.

This is not Prompt Caching or KV Cache. Prompt Caching targets model input prefix reuse, KV Cache targets attention state reuse during inference, while Tool Result Caching targets external tool execution results. It’s an engineering layer capability of the Agent Runtime / Tool Gateway / LLM Application Backend.

Core Principle: Insert a Result Cache Layer Between Tool Calls and External Tools

A minimal closed loop looks like this:

1. The model outputs a tool call, e.g., get_weather({"city":"Tokyo"}).
2. The Agent Runtime normalizes the tool name and arguments.
3. A cache key is generated, e.g., tenant:user:tool:version:normalized_args:auth_scope.
4. The cache is queried.
5. Hit and fresh: the cached result is returned directly to the model.
6. Miss, expired, or non-cacheable: the real tool call is executed.
7. After a successful tool call, the result, source, time, TTL, version, and audit info are written to the cache.
8. The model generates the final response based on the tool result.

The key point of this design isn’t simply “add a Redis.” It’s about answering five questions:

• What can be cached: Read-only queries, stable data, results with high short-term repetition.
• How long to cache: Different tools, fields, and business scenarios require different TTLs.
• Who can reuse: Tenant, user, permissions, and data scope must be part of the cache boundary.
• How to invalidate: TTL, event notifications, version upgrades, and manual purging must all be supported.
• How to observe: Hit rate, saved calls, staleness rate, error rate, and cost savings must be visible.

Cache Key Design: Don’t Just Use Tool Name and Arguments

Many cache incidents happen not because there’s no cache, but because the cache key is too coarse. For example:

// ❌ Dangerous practice
const badKey = tool_name + JSON.stringify(args);

This works in demos, but in a multi-tenant production environment, problems arise:

• Two users both call get_order_status({orderId:"123"}), but do they both have permission to view the same order?
• The same search_policy({keyword:"surrender"}), but different regions, product lines, languages, or release times—are they the same?
• The same get_price({sku:"A"}), but differentiated by channel, currency, membership level, and time window?

A production cache key should include at least these dimensions:

Dimension	Description
tenant_id	Tenant identifier
user_or_service_scope	User or service authorization scope
tool_name	Tool name
tool_version	Tool version
normalized_arguments	Normalized arguments
authorization_scope	Authorization scope hash
data_region_or_locale	Data region or locale
freshness_policy_id	Freshness policy ID
schema_version	Data schema version

Argument normalization should do three things:

• Sort JSON keys to avoid {a:1,b:2} and {b:2,a:1} producing different keys.
• Remove meaningless fields like traceId, requestId, UI source.
• Bucket time parameters, e.g., convert “last 24 hours” to a specific time range.

Recommended generation approach:

import crypto from "node:crypto";

function stableStringify(value: unknown): string {
  if (Array.isArray(value)) {
    return `[${value.map(stableStringify).join(",")}]`;
  }
  if (value && typeof value === "object") {
    return `{${Object.entries(value as Record<string, unknown>)
      .filter(([key]) => !["traceId", "requestId"].includes(key))
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([key, val]) => `${JSON.stringify(key)}:${stableStringify(val)}`)
      .join(",")}}`;
  }
  return JSON.stringify(value);
}

function buildToolCacheKey(input: {
  tenantId: string;
  authScopeHash: string;
  toolName: string;
  toolVersion: string;
  args: unknown;
  freshnessPolicyId: string;
  schemaVersion: string;
}) {
  const payload = stableStringify(input.args);
  const digest = crypto.createHash("sha256").update(payload).digest("hex");
  return [
    "tool-result",
    input.tenantId,
    input.authScopeHash,
    input.toolName,
    input.toolVersion,
    input.freshnessPolicyId,
    input.schemaVersion,
    digest,
  ].join(":");
}

Security Note: authScopeHash should not directly contain plaintext sensitive information like email, phone number, or ID number. You can hash authorization scopes, roles, resource sets, and data domains, but don’t expose personal sensitive data as part of the cache key in Redis keys, logs, or monitoring systems.

TTL Is Not a One-Size-Fits-All Configuration; It’s Part of the Tool Contract

Cache TTL should be bound to the tool’s semantics, not set by the infrastructure team with a single default value. It can be layered by data type:

Tool Type	Example	Recommended TTL	Invalidation Method
Static rules	Help docs, policy explanations, field dictionaries	Hours to days	Document release events, version numbers
Near real-time info	Weather, exchange rates, public prices	Seconds to minutes	TTL + background refresh
User state	Order status, ticket status, account balance	Seconds or not shared	Business events + permission checks
High-risk write operations	Payment, refund, send email, create order	No result caching	Idempotency keys and audit
Search/retrieval results	Web search, internal knowledge base retrieval	Minutes	Index version + query hash

HTTP caching standard RFC 9111 emphasizes that caching reduces latency and network overhead by reusing existing responses, but it must decide based on freshness, validation, cache key, and security constraints. This concept can be migrated to Agent tool result caching: the cache is not a permanent answer repository, but a result reuse mechanism with freshness, permission, and version boundaries.

Idempotency Keys and Result Caching Are Different

Many teams confuse idempotency with caching. They are related, but not the same.

• Idempotency keys solve the problem of “the same write operation being executed multiple times due to timeout, retry, or network jitter.” Stripe’s API documentation explains that you can use an idempotency key when creating or updating objects; subsequent requests with the same key return the status code and response body saved from the first request, preventing duplicate creation or updates.
• Result caching solves the problem of “whether multiple equivalent read requests can reuse the same known result.”

For Agents, the rules can be:

• Read tools: Consider Tool Result Caching.
• Write tools: Default to no result caching; use idempotency keys, approval, audit, and compensation.
• Read-write mixed tools: Split into read preview and commit action, governed separately.

For example, calculate_shipping_fee is a read-type calculation and can be cached; create_shipping_order is a write-type action and should not reuse old results just because parameters are the same—use idempotency keys to prevent duplicate creation.

Engineering Implementation: Recommend Caching as a Tool Gateway Capability

If each tool implements caching independently, you’ll end up with inconsistent TTLs, unmeasurable hit rates, chaotic permission boundaries, and an inability to invalidate uniformly. A safer approach is to add a Tool Gateway layer between the Agent Runtime and the real tools.

A simplified execution flow is as follows:

type ToolCall = {
  tenantId: string;
  userId: string;
  toolName: string;
  toolVersion: string;
  args: unknown;
  authScopeHash: string;
};

type ToolPolicy = {
  cacheable: boolean;
  ttlSeconds: number;
  allowStaleOnError: boolean;
  negativeCacheSeconds?: number;
  schemaVersion: string;
  freshnessPolicyId: string;
};

async function executeToolWithCache(call: ToolCall) {
  const policy = await loadToolPolicy(call.toolName, call.toolVersion);
  if (!policy.cacheable) {
    return executeRealTool(call);
  }
  const key = buildToolCacheKey({
    tenantId: call.tenantId,
    authScopeHash: call.authScopeHash,
    toolName: call.toolName,
    toolVersion: call.toolVersion,
    args: call.args,
    freshnessPolicyId: policy.freshnessPolicyId,
    schemaVersion: policy.schemaVersion,
  });
  const cached = await cache.get(key);
  if (cached && !cached.isExpired) {
    recordMetric("tool_cache_hit", call.toolName);
    return {
      ...cached.value,
      _cache: {
        hit: true,
        key,
        storedAt: cached.storedAt,
        expiresAt: cached.expiresAt,
      },
    };
  }
  try {
    const result = await executeRealTool(call);
    await cache.set(key, result, policy.ttlSeconds);
    recordMetric("tool_cache_miss", call.toolName);
    return { ...result, _cache: { hit: false, key } };
  } catch (error) {
    if (cached && policy.allowStaleOnError) {
      recordMetric("tool_cache_stale_served", call.toolName);
      return {
        ...cached.value,
        _cache: { hit: true, stale: true, reason: "origin_error" },
      };
    }
    throw error;
  }
}

This example intentionally retains _cache metadata. When returning to the model, you can pass only business fields; but logs, traces, and monitoring systems must record cache status, otherwise troubleshooting later will be very difficult.

Applicable Scenarios

Tool Result Caching is most suitable for the following scenarios:

1. High External API Costs

Examples: Web Search, geocoding, exchange rates, weather, business registration info, logistics tracking, financial market data, OCR results. As long as results are reusable within a short time, caching reduces call fees and rate limit pressure.

2. High Query Repetition

Customer service, operations, sales assistants, and internal knowledge base Q&A often generate many repeated tool calls. User questions may differ, but after normalization, tool parameters can be identical.

3. Tool Latency Significantly Higher Than Model Inference

If the model can generate a tool call in 800ms, but the external API averages 2 seconds with a P95 of 8 seconds, cache hits will dramatically improve user experience.

4. Downstream Systems Need Protection

Legacy core systems, CRMs, ERPs, risk control systems, insurance cores, and ticketing systems typically don’t want to be hit directly by high-concurrency Agent traffic. Caching can act as a peak-shaving layer, but it cannot replace permission control and rate limiting.

Common Misconceptions

Misconception 1: Higher Cache Hit Rate Is Always Better

No. A high hit rate with stale answers can harm the business. For data like prices, inventory, and order status, you should also look at hit rate, stale serve rate, freshness violation, and user correction rate.

Misconception 2: Cached Results Can Be Shared Across Tenants

Not by default. Unless the data is public, has no permission differences, and has no personalization differences—e.g., public weather, public documents, public exchange rates. In multi-tenant systems, tenant, user permissions, and data domains must be part of the cache boundary.

Misconception 3: Same Tool Parameters Always Produce the Same Result

Many tools have implicit context: user role, region, language, time, product line, channel, canary version, model version, index version. If the cache key doesn’t include these factors, incorrect reuse can occur.

Misconception 4: Caching Can Solve All Tool Call Costs

Caching only solves “reusable read results.” For high-risk write operations, what’s needed is idempotency, approval, rollback, audit, and compensation, not ordinary caching.

Misconception 5: The Model Doesn’t Need to Know About Cache Status

Not necessarily. For some scenarios, the model needs to know whether a result is from a real-time query or a cache. For example, when answering “Was this just retrieved?”, if the tool result came from a 10-minute-old cache, it should be able to indicate the time. It’s recommended to pass metadata like observed_at, source, ttl, and is_cached to the model in a controlled manner.

Production Launch Checklist

Before going live, at least check the following items:

1. Does each tool have a cacheable flag?
2. Does the cache key include tenant, permissions, tool version, schema version, and argument hash?
3. Is caching of write-type tool results prohibited?
4. Are sensitive fields masked or excluded from the cache?
5. Is forced invalidation supported per tool, tenant, and version?
6. Is there a negative cache to prevent downstream errors from being amplified indefinitely?
7. Are timeouts, 4xx, 5xx, and empty business results differentiated?
8. Is stale-on-error supported, and are the tools that can use it clearly defined?
9. Are hit, miss, stale, bypass, evict, and invalidate metrics recorded?
10. Are cache key hash, TTL, storedAt, and expiresAt shown in traces?
11. Is there an emergency cache pollution cleanup script?
12. Is there a small-traffic canary to confirm that cached results don’t change business semantics?

Summary

Tool Result Caching is an easily overlooked but highly impactful capability in Agent engineering. It’s not simply wrapping tool calls with Redis; it’s a systematic trade-off between cache key design, TTL strategy, tenant isolation, invalidation mechanisms, observability systems, and security boundaries. Proper cache design allows Agents to significantly reduce external API costs, protect downstream systems, and improve tail latency while maintaining accuracy—and these are the critical steps for a production-grade Agent to move from demo to large-scale deployment.

References

• OpenAI Function Calling Guide
• Anthropic Tool Use with Claude
• Stripe Idempotent Requests
• RFC 9111 HTTP Caching
• OWASP Top 10 for LLM Applications 2025