Tool Contract Testing in Production: Governing Function Call Schema Evolution with Contract Cases

Background: Agent Failures Aren’t Always a Model Problem

In large language model applications, Function Calling and Tool Use are typically seen as the standard way to connect models to real business systems. The model selects a tool based on user intent, generates structured parameters, the application layer executes the external API, and then returns the tool result to the model for further reasoning.

This pipeline seems straightforward, but a subtle problem often emerges in production: The model isn’t obviously hallucinating, the tool isn’t down, the call format even conforms to JSON Schema, yet the business outcome still fails.

Common scenarios include:

• A new required field is added to a tool, but the old Agent prompt doesn’t generate it.
• Parameter enums change from standard / premium to basic / pro, but old call paths still pass the old values.
• The field name stays the same, but its semantics change from “city name” to “city ID”.
• The tool’s return structure gains a nesting level, but subsequent reasoning prompts still read from the old path.
• Error codes shift from business errors to platform errors, breaking the Agent’s retry / fallback strategy.
• A tool is shared by multiple Agents, and a small change by one team breaks an implicit dependency of another team.

The root cause isn’t that “the model can’t call tools,” but rather that tool contracts aren’t managed with engineering discipline. Schema is only part of the contract. The real production contract also includes call intent, parameter semantics, default values, error handling, return fields, permission boundaries, cost constraints, and compatibility commitments.

Therefore, the Agent toolchain shouldn’t just test “can it generate valid JSON?” It needs Tool Contract Testing: using executable contract cases to continuously verify that the tool schema, Agent prompt, tool implementation, and downstream business interfaces still satisfy the shared agreement.

Core Principle: Treat Tool Definitions as Versioned API Contracts

OpenAI’s function calling documentation defines function tools as tools described by a JSON Schema for input parameters; strict: true makes function calls more reliably conform to the function schema and requires objects to disallow extra fields and declare all properties as required. Anthropic’s Tool Use documentation also clarifies that developers need to pass an input_schema for custom tools, and the model returns a tool_use which the application code executes and returns a tool_result.

This shows that tool calling isn’t just a prompting trick; it’s a cross-boundary interface protocol:

User Intent → Agent Prompt → Tool Selection → Tool Arguments
    → Tool Runtime → External API / Database → Tool Result → Agent Continuation

Wherever there’s a cross-boundary interface, a contract is needed. Pact defines contract testing as testing integration points by checking that each application, in isolation, sends or receives messages that conform to a shared understanding. Google AIP-180 also emphasizes that an API is fundamentally a contract with its users, who write production code against it, so compatible and incompatible changes must be distinguished.

Applied to Agent tool calling, the contract includes at least four layers:

1. Schema Contract

The schema contract describes what parameters a tool can accept. It includes field names, types, enums, nested structures, required status, whether extra fields are allowed, field descriptions, and default values.

Example:

{
  "name": "create_refund_request",
  "description": "Create a refund request for a paid order.",
  "strict": true,
  "parameters": {
    "type": "object",
    "additionalProperties": false,
    "properties": {
      "order_id": {
        "type": "string",
        "description": "The internal order identifier."
      },
      "reason_code": {
        "type": "string",
        "enum": ["duplicate_payment", "customer_request", "service_failure"]
      },
      "amount_cents": {
        "type": ["integer", "null"],
        "description": "Refund amount in cents. Null means full refund."
      }
    },
    "required": ["order_id", "reason_code", "amount_cents"]
  }
}

Such definitions solve the problem of “unstable parameter structure,” but they can’t solve semantic compatibility issues on their own. For example, whether amount_cents allows partial refunds, whether order_id can be an external order number, or whether reason_code affects the approval workflow — these are things JSON Schema alone cannot express.

2. Semantic Contract

The semantic contract describes what a field truly represents and which input combinations are valid.

For example:

• amount_cents = null means a full refund, not a zero-dollar refund.
• service_failure must be accompanied by a ticket ID, otherwise it goes to manual review.
• Settled orders can only initiate a refund request, not a direct refund.
• Orders older than 30 days require higher permissions.
• When the tool returns status=pending_review, the Agent must not tell the user “refund successful.”

These aspects need to be expressed through contract cases, not just field descriptions.

3. Behavioral Contract

The behavioral contract describes how the tool should respond in different states. It covers success paths, business failures, permission failures, rate limiting, timeouts, idempotency, and retries.

For example:

case_id: refund_partial_requires_amount
intent: "The user requests a partial refund."
input:
  order_id: "ord_123"
  reason_code: "customer_request"
  amount_cents: 5000
expected:
  tool_status: "created"
  agent_next_action: "tell_user_request_submitted"
must_not_say:
  - "The refund has been credited."

This kind of case validates both the tool’s return and how the Agent interprets the tool result.

4. Evolution Contract

The evolution contract defines which changes can go live directly, which require a version bump, and which require dual writes or canary testing.

Tool changes can be categorized into three types:

Category	Typical Changes	Handling Strategy
Compatible	Adding optional fields, adding return fields (old consumers can ignore), relaxing field length or value ranges, adding non-default capabilities, adding error codes while preserving old mappings	Can go live directly, bump minor version
High-Risk	Adding required fields, deleting fields, renaming fields, changing field types, deleting or renaming enum values, changing default values, changing semantics of same-named fields, changing return structure paths, changing error code categories	Requires contract replay verification, CI gating
Must Bump Major	New parameters that old Agents cannot auto-fill, changes to business semantics that alter user-visible results, changes to parameters affecting permissions/security/fund flows, changes affecting tools shared by multiple Agents, structural changes that break continuation of historical sessions	Bump major version or run dual track

Engineering Implementation: From Schema Diff to Replay Testing

Tool Contract Testing doesn’t need to start as a complex platform. A more practical approach is to first establish four minimal loops: Schema Registry, Contract Cases, Replay Runner, and Release Gate.

1. Establish a Tool Schema Registry

Every tool should have a stable ID and version number, rather than having a JSON snippet scattered in the code.

Suggested fields:

tool_id: refund.create_request
version: 2.1.0
owner: payment-platform
runtime: internal-http
schema_file: schemas/refund.create_request.v2.1.0.json
changelog: docs/tools/refund.create_request/changelog.md
compatibility:
  compatible_with:
    - 2.0.x
  deprecated_versions:
    - 1.x
risk_level: high

tool_id should not change frequently. The version number expresses contract changes; don’t cram all changes into a single “latest version” of the tool.

2. Automate Schema Diffing

Every time a tool schema changes, CI should automatically diff the old and new versions and provide a risk assessment.

Pseudo-code:

def classify_schema_change(old_schema, new_schema):
    changes = diff_json_schema(old_schema, new_schema)
    risks = []
    for change in changes:
        if change.kind in ["remove_property", "rename_property", "change_type"]:
            risks.append("breaking")
        elif change.kind == "add_required_property":
            risks.append("breaking")
        elif change.kind == "narrow_enum":
            risks.append("breaking")
        elif change.kind == "add_optional_property":
            risks.append("compatible")
        elif change.kind == "add_response_field":
            risks.append("usually_compatible")
        else:
            risks.append("needs_review")
    return max_risk(risks)

This step can’t fully replace manual review, but it can catch obvious breaking changes early.

3. Cover Real Agent Calls with Contract Cases

Contract cases shouldn’t just test “tool parameters are valid.” It’s more valuable to cover the path from real user intent to tool invocation.

A recommended executable contract case includes:

case_id: refund_pending_review_not_success
source: production_replay
agent: customer_service_agent
user_intent: "The user requests a refund, but the order exceeds the auto-refund period."
tool:
  id: refund.create_request
  version: 2.1.0
expected_tool_arguments:
  order_id: "ord_123"
  reason_code: "customer_request"
  amount_cents: null
mock_tool_result:
  status: "pending_review"
  review_eta_hours: 24
expected_agent_behavior:
  must_include:
    - "Refund request has been submitted."
    - "Requires manual review."
  must_not_include:
    - "Refund successful."
    - "The funds have been returned to the original account."

Here, the test isn’t about whether the model “can speak Chinese,” but whether the Agent correctly interprets the tool’s return status into the right business meaning.

4. Build a Replay Test Runner

Before a release, the new schema, new prompt, new tool implementation, and a batch of historical cases should be replayed in an isolated environment.

Three types of replay are recommended:

• Golden Cases: Core business paths that must all pass.
• Regression Cases: Historical production issues that, if they fail, must block the release.
• Shadow Cases: Anonymized samples from recent production traffic, used to discover unknown risks.

The replay runner’s output shouldn’t just be pass/fail; it should also provide a change summary:

Tool: refund.create_request v2.1.0 -> v2.2.0
Schema diff: add required field approval_channel
Risk: breaking
Replay result:
  - golden:   48 / 50 passed
  - regression: 31 / 31 passed
  - shadow:   894 / 1000 passed
Top failures:
  1. Missing approval_channel in old prompt path: 72 cases
  2. Agent misread pending_review as success: 19 cases
  3. Enum value deprecated: 15 cases
Decision: block release

This kind of report is more actionable for an engineering team than a simple “model evaluation score dropped by 2%.“

5. Release Gate: Tool Changes Must Pass a Checkpoint

Tool releases shouldn’t just be a code merge by the tool provider. For tools depended on by Agents, the release process should include a Release Gate:

Schema Diff → Compatibility Classification → Contract Replay
    → Owner Review → Canary Agent Traffic → Version Promotion
    → Deprecation Window

High-risk tools should also require manual sign-off, such as for payments, policies, orders, permissions, email sending, database writes, and file deletion.

Applicable Scenarios

Tool Contract Testing is particularly suitable for the following scenarios:

1. Tools shared by multiple Agents. A CRM query tool might be used by a sales assistant, a customer service assistant, and an operations assistant. A small change to a tool field might only surface a problem in one Agent.

2. Tools connected to real business systems. Whenever money, orders, contracts, permissions, notifications, approvals, or database write operations are involved, you can’t rely on “the model will probably generate the correct parameters.”

3. Tool schemas change frequently. Early Agent projects often rapidly add fields, adjust enums, and modify error structures. Without contract testing, hidden technical debt accumulates quickly.

4. Prompts and tools are maintained by different teams. The prompt team assumes the tool semantics are stable, the platform team assumes the schema is compatible, and the business team assumes the field meaning hasn’t changed — until a production issue surfaces.

5. Support for historical session recovery is needed. Agent sessions may resume across days. If only the latest schema is kept, intermediate states from old sessions may become unrecoverable.

Common Misconceptions

Misconception 1: JSON Schema Equals a Contract

JSON Schema can only express a part of structural constraints. It cannot fully express business state, permissions, idempotency, costs, error recovery, or user-visible promises.

The correct approach is: Schema handles structure; contract cases handle semantics and behavior.

Misconception 2: Strict Mode Can Replace Testing

strict mode reduces formatting errors, but it cannot determine “whether this tool should be called,” “whether execution should continue after the call,” or “how the tool result should be interpreted.” Production testing still needs to cover tool selection, parameter generation, tool execution, and result interpretation.

Misconception 3: Only Test the Latest Prompt and Latest Tool

Real production environments often have old sessions, old caches, old clients, old Agent configurations, and canary users. Contract testing must cover multiple version combinations, not just the latest version.

Misconception 4: Attribute All Failures to the Model

Many Agent failures aren’t the model’s fault; they’re caused by tool contract changes not being captured by the release process. Attributing failures to the model leads teams to repeatedly tweak prompts while ignoring the real risk of interface evolution.

Misconception 5: Contract Testing is Only for Microservices, Not LLM Agents

Agent tool calling is fundamentally cross-system integration. The difference is that the traditional consumer is code, while the Agent consumer includes prompts, model behavior, tool schemas, and tool result interpretation. Therefore, it needs contract testing more, not less.

Release Checklist

Tool Definition Check

• Does the tool have a stable tool_id?
• Is the schema under version control?
• Are the owner, risk_level, and deprecated version marked?
• Are unnecessary extra fields disabled?
• Do field descriptions specify units, default values, permissions, and boundary conditions?
• Do enums have clear semantics, not just short strings?
• Do error codes have stable categories?

Compatibility Check

• Are new required fields being added?
• Are fields being deleted?
• Are fields being renamed or having their types changed?
• Are enums being narrowed?
• Are default values changing?
• Are return structure paths changing?
• Are error code semantics changing?
• Will old Agent prompts or historical sessions be affected?

Contract Case Check

• Are core success paths covered?
• Are business failure paths covered?
• Are permission failures covered?
• Are timeouts, rate limiting, and retries covered?
• Are historical production defects covered?
• Are high-risk parameter combinations covered?
• Is tool result interpretation covered?

Release Check

• Has the Schema Diff passed?
• Has the Contract Replay passed?
• Is canary traffic showing no anomalies?
• Is a rollback to the old version configured?
• Is the old tool version retained for a period?
• Has a changelog been published to Agent application teams?
• Is it recorded who approved the high-risk change?

A Practical Directory Structure

agent-tools/
  refund.create_request/
    schemas/
      v2.1.0.json
      v2.2.0.json
    contracts/
      golden/
        refund_full_success.yaml
        refund_pending_review.yaml
      regression/
        old_prompt_missing_reason_code.yaml
      shadow/
        2026-07-traffic-sample.yaml
    changelog.md
    owner.yaml
    release-policy.yaml

Each tool groups its schema, contract cases, change log, and release policy together. The benefit is that when a tool changes, CI can directly identify the affected contracts without needing a global search for prompts and test files.

Summary

The stability of Agent tool calling depends on more than just whether the model can generate JSON. What truly determines production reliability is whether the tool contract can be versioned, tested, and integrated into the release process.

The core idea of Tool Contract Testing is simple: Treat the tool schema as an API, the Agent prompt as a consumer, the tool implementation as a provider, and real business paths as executable contracts. Before every tool change, run a Schema Diff, then a Contract Replay, then a canary release.

This approach won’t make Agents “absolutely reliable,” but it will surface a large number of hidden schema evolution risks early, preventing teams from discovering only after a release that old prompts, old sessions, and old tool call paths have been broken.

Key References

• OpenAI Function Calling Guide: https://platform.openai.com/docs/guides/function-calling
• Anthropic Tool Use Overview: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/overview
• Pact Documentation: https://docs.pact.io/
• Google AIP-180 Backwards Compatibility: https://google.aip.dev/180
• Do Large Language Models Respect Contracts? Evaluating and Enforcing Contract-Adherence in Code Generation: https://arxiv.org/abs/2510.12047