LLM Multi-Provider API Adapter Production Practices: Reducing Migration Costs with Capability Matrices, Parameter Normalization, and Response Contracts

Background: The More Model Providers You Have, the Harder the “Contract Drift”

When many teams first integrate large language models, their code has only one provider, one model, and one request format. Once the business is running, more complex requirements emerge: some tasks need OpenAI, some need Claude, some need Gemini, some clients require private cloud or local models, and some scenarios need to switch models based on price, latency, context window, vision capabilities, or structured output capabilities.

At this point, the easiest mistake is treating “switching an SDK” as a model migration. On the surface, all calls are messages -> response; but in production, differences appear in every detail:

• Where is the system instruction placed?
• How are tool calls returned?
• How do streaming events end?
• Is the usage field complete?
• How is a refusal expressed?
• Can structured output still be parsed on failure?
• Is the meaning of the max token parameter consistent?

OpenAI’s Responses API supports text, image, and file inputs, can use built-in tools, MCP tools, and custom function calls, and supports streaming output. Anthropic’s Messages API emphasizes direct model prompting with a custom agent loop, with stop_reason and usage in responses. Gemini’s generateContent uses structures like contents[], tools[], system_instruction, response_schema, and usageMetadata. LiteLLM provides a unified interface, wrapping multiple providers into an OpenAI-like format. These references all seem to be about “calling a model,” but they reveal an engineering truth: a unified entry point does not mean unified semantics.

Therefore, a maintainable multi-provider LLM integration layer should not just be an SDK wrapper. It should be a clear Provider Adapter Contract: upstream business logic depends only on the platform’s internal contract; downstream provider differences are explicitly declared, transformed, degraded, and verified through replay by the adapter layer.

Core Principle: Define the Internal Contract First, Then Connect Provider APIs

1. Keep the Internal Request Model Stable

Don’t let business code directly perceive the raw request bodies of OpenAI, Anthropic, Gemini, or some open-source inference service. The platform should define a relatively stable internal request model, for example:

type LlmRequest = {
  requestId: string;
  tenantId: string;
  taskType: "chat" | "extract" | "tool_agent" | "vision" | "classification";
  modelPolicy: {
    preferredModel?: string;
    requiredCapabilities: Capability[];
    fallbackAllowed: boolean;
  };
  instructions?: string;
  messages: InternalMessage[];
  tools?: InternalToolSpec[];
  responseContract?: ResponseContract;
  generation: {
    maxOutputTokens?: number;
    temperature?: number;
    topP?: number;
    stop?: string[];
  };
  metadata: Record<string, string>;
};

This model isn’t about “abstract elegance”; it’s about decoupling business logic from provider specifics. The business only expresses what it needs: does it require tool calling? Must it be JSON? Does it need vision input? Is fallback allowed? What’s the maximum output token count? The adapter handles how to construct the request body for a given provider.

2. Make the Capability Matrix Explicit, Don’t Discover It Through Runtime Errors

The most important table for multi-provider integration isn’t a routing table; it’s the capability matrix. Every model should declare what it supports, what it doesn’t, and to what extent.

At a minimum, record these dimensions:

Capability Dimension	What to Record
Input Types	text, image, audio, video, file, URL, PDF, etc.
Output Types	text, JSON schema, tool call, multi-part response
Tool Calling	Supports parallel tool calls, strict tool use, server tools, MCP
Context Window	Max input tokens, max output tokens, supports automatic truncation
Streaming Output	Supports SSE, chunk format, usage returned with stream
Safety & Refusal	Refusal, block reason, safety rating, stop reason expression
Caching & State	Supports prompt cache, previous response, conversation state
Cost Statistics	Usage field granularity, cached tokens, reasoning tokens visible
Provider Limitations	Unsupported sampling parameters, special model constraints, regional or service tier limits

This table must be readable at runtime, not just in documentation. Perform capability validation before making a call: if the business requires a JSON schema but the target model only supports plain text, reject the request or switch models before the call, not when parsing fails in production.

Parameter Normalization: Don’t Assume Same-Named Parameters Have the Same Semantics

temperature, top_p, and top_k

Many models have temperature, but whether it’s supported, its default value, recommended range, and how it combines with other sampling parameters are not all the same. Anthropic’s documentation explicitly states that some new Claude Opus models do not support non-default temperature, top_p, or top_k, and setting non-default values will return a 400 error. This shows that parameter normalization can’t just be about renaming fields.

A safer approach is:

type ParamPolicy = {
  supported: boolean;
  providerField?: string;
  defaultValue?: unknown;
  allowedRange?: [number, number];
  unsupportedBehavior: "drop" | "fail" | "warn_and_drop";
};

For parameters with strong business semantics, like response_schema, tool_choice, or max_output_tokens, don’t silently drop them. For weakly controlling parameters, like some sampling parameters, you can degrade according to policy, but you must log it in the trace: what the original parameter was, what was actually sent to the provider, and why it was removed.

The Edge of max token

max_tokens, max_output_tokens, and max_completion_tokens all seem to limit output length, but in different APIs they might include visible output, reasoning tokens, or other internal tokens. A production system shouldn’t just record a single max_tokens; it should be broken down into:

type TokenBudget = {
  maxInputTokens?: number;
  maxVisibleOutputTokens?: number;
  maxReasoningTokens?: number;
  hardContextLimit?: number;
  truncationPolicy: "fail" | "truncate_oldest" | "summarize_then_call";
};

The adapter layer is responsible for mapping the internal budget to provider fields. If a provider can’t express a certain budget, like being unable to control reasoning tokens independently, it must be flagged in the capability matrix, and cost and latency should be closely monitored during canary testing.

Response Contracts: Parsing Results is More Prone to Failure Than Sending Requests

Unified Response Object

It’s recommended that the platform doesn’t return raw provider responses directly, but instead returns a unified LlmResult:

type LlmResult = {
  requestId: string;
  provider: string;
  model: string;
  status: "completed" | "refused" | "blocked" | "tool_call" | "length_exceeded" | "failed";
  content: Array<
    | { type: "text"; text: string }
    | { type: "json"; value: unknown; raw: string }
    | { type: "tool_call"; name: string; arguments: unknown; callId: string }
  >;
  finish: {
    reason: "stop" | "length" | "tool_call" | "content_filter" | "error" | "unknown";
    providerReason?: string;
  };
  usage?: {
    inputTokens?: number;
    outputTokens?: number;
    reasoningTokens?: number;
    cachedTokens?: number;
    totalTokens?: number;
  };
  rawRef: string;
};

The most critical part here is preserving providerReason and rawRef. A unified contract is convenient for business logic, but troubleshooting must allow tracing back to the original response. Otherwise, after a migration, if there are “sporadic empty results,” “tool calls parsed as text,” or “missing JSON fields,” the platform team will struggle to pinpoint whether the issue is in the model, the adapter layer, the call parameters, or the business parser.

Don’t Crudely Merge stop reasons

Different providers have “stop reasons,” but their semantics aren’t identical. Anthropic’s example responses include stop_reason and usage; Gemini responses have candidate results, finishReason, promptFeedback, safetyRatings, and usageMetadata; OpenAI’s Responses API has response status, output items, streaming events, usage, and tool call objects.

The adapter layer can map these to an internal enum, but it must not lose the original value. The recommended approach is:

{
  "finish": {
    "reason": "content_filter",
    "providerReason": "SAFETY",
    "providerPath": "promptFeedback.blockReason"
  }
}

This way, business logic only needs to check reason, while platform troubleshooting can see the real source.

Tool Call Adaptation: Don’t Just Unify the Schema, Unify the Lifecycle

Tool calling is the most error-prone part of multi-provider adaptation. On the surface, it’s all function names and arguments, but the actual lifecycle differs:

1. How does the model declare it wants to call a tool?
2. Does it support parallel tool calls?
3. Is the tool call ID stable?
4. How are tool results fed back?
5. How are tool errors communicated back to the model?
6. Are tool call arguments streamed in chunks?
7. Does the model continue generating final text after a tool call?

Therefore, the tool adapter layer needs to model a tool call as a state machine, not just a simple field transformation:

type ToolCallState =
  | { state: "requested"; callId: string; name: string; argsText: string }
  | { state: "parsed"; callId: string; name: string; args: unknown }
  | { state: "executing"; callId: string }
  | { state: "succeeded"; callId: string; result: unknown }
  | { state: "failed"; callId: string; errorType: string; retryable: boolean }
  | { state: "returned_to_model"; callId: string };

When migrating models, tool call testing shouldn’t just check “if the call succeeds.” You also need to test:

• Changes in argument order
• Field default values
• Missing fields
• Empty arrays
• Invalid enums
• Consecutive multiple tool calls
• Model refusing to call a tool
• Tool return being too long
• Tool error retries

Streaming Output Adaptation: Unify Chunks, Not Events, or Risk Incidents

Streaming output isn’t just “returning text in chunks.” A production system must handle at least:

• Text deltas
• Tool call deltas
• Refusal or safety blocks
• Finish events
• Usage events
• Provider error events
• Client cancellation
• Partial results after network interruption

It’s recommended to unify internally into an event stream:

type LlmStreamEvent =
  | { type: "text_delta"; text: string }
  | { type: "tool_call_delta"; callId: string; name?: string; argsDelta?: string }
  | { type: "usage"; usage: LlmResult["usage"] }
  | { type: "finish"; finish: LlmResult["finish"] }
  | { type: "error"; code: string; retryable: boolean; providerError?: unknown };

The key point here isn’t a pretty format; it’s that the web frontend, logging system, Agent Runtime, billing system, and auditing system all see the same set of events. Otherwise, when you switch providers, the frontend might still display text, but tool calls, usage statistics, abnormal terminations, and request cancellations could silently become inaccurate.

Migration Governance: Replay Testing is Mandatory Before Going Live

Before a multi-provider adapter layer goes live, the most important thing isn’t unit tests; it’s request replay.

It’s recommended to sample and anonymize real production requests to build a migration replay set:

Replay Category	Sample Requirements
Normal Conversation	Multi-turn history, system instruction, long input
JSON Output	Required fields, nested arrays, enums, null values
Tool Calling	Single tool, multiple tools, parallel tools, tool errors
Safety Boundaries	Refusals, content filtering, sensitive input, low-risk false positive samples
Boundary Exceedance	Near context window, output too long, truncation policy
Streaming Output	User interruption, network disconnection, tool argument chunking
Cost Boundaries	High token input, cache hits, reasoning toggle

For each sample, record the old model result, the new model result, the adapted internal response, the parsed result, and the final business-side result. Whether a migration passes shouldn’t just depend on text similarity; it should depend on contract stability: are status codes consistent? Is JSON parseable? Are fields complete? Are tool calls executable? Are refusals explainable? Can usage be billed?

Engineering Implementation: A Minimal Architecture for an Adapter Layer

A maintainable LLM API adapter layer can be broken down into five layers:

1. Capability Registry

Responsible for maintaining providers, models, and the capability matrix. When a model is upgraded, the capability declaration must be updated before traffic is routed to it.

2. Request Normalizer

Transforms business requests into internal standard requests, performs basic validation, token estimation, default parameter filling, and trace metadata injection.

3. Provider Adapter

One adapter per provider, responsible for mapping internal requests to real API requests and mapping responses back to the unified contract.

4. Replay & Contract Test

Saves typical request samples and periodically runs replay tests against adapters, model versions, and provider API changes.

5. Runtime Ledger

Records the original parameters, transformed parameters, provider response summary, usage, cost, errors, fallbacks, retries, and final business status for each request.

A simplified directory structure could be:

llm-platform/
  adapters/
    openai-responses.ts
    anthropic-messages.ts
    gemini-generate-content.ts
    local-openai-compatible.ts
  contracts/
    request.ts
    response.ts
    stream-events.ts
    capability.ts
  registry/
    models.yaml
    provider-capabilities.yaml
  replay/
    fixtures/
    expected/
    runner.ts
  ledger/
    request-ledger.ts

Common Pitfalls

Pitfall 1: A Unified SDK Equals Unified Capabilities

Tools like LiteLLM can significantly reduce integration costs and provide a unified interface, output format, retries, fallback, and cost tracking. However, production teams still need to maintain their own business contracts. A unified SDK solves “how to call,” not “whether this model satisfies the current business semantics.”

Pitfall 2: Only Testing the Happy Path

The most problematic areas during model migration are usually not standard Q&A, but error paths: content filtering, length limits, illegal tool parameters, truncated JSON, streaming interruptions, and missing usage data. Adapter layer testing must cover failure paths.

Pitfall 3: Silently Dropping Unsupported Parameters

Some parameters can be degraded, others cannot. For example, temperature might be removable with a log entry, but parameters like response_schema, tool_choice, max_output_tokens, or safety_identifier should fail explicitly or trigger a model switch if they cannot be mapped.

Pitfall 4: Only Saving the Unified Response, Not the Original Reference

A unified response is convenient for business logic, but troubleshooting requires original evidence. It’s recommended to not store the full original request and response directly; at a minimum, save a safely anonymized summary, object IDs, provider request IDs, error codes, usage, finish reason, and the adapter version.

Go-Live Checklist

Before going live, confirm each item:

• Capability Matrix covers all target models, distinguishing “supported,” “partially supported,” and “not supported.”
• Parameter Mapping has unit tests, especially for tokens, temperature, tool_choice, response_schema, and stream.
• Response Contract covers text, JSON, tool calls, refusals, blocks, length limits, and errors.
• Streaming Events cover text deltas, tool deltas, usage, finish, errors, and cancellation.
• Replay Set contains real anonymized samples covering high-frequency and boundary paths.
• Canary Strategy supports gradual rollout by tenant, task type, model capability, and cost budget.
• Rollback Strategy can not only switch back to the old provider but also handle already-stored new format responses.
• Request Ledger records adapter version, model version, parameter transformations, usage, cost, errors, and fallbacks.
• Monitoring Metrics include at least success rate, parse failure rate, structured output failure rate, tool call failure rate, P95/P99 latency, average token cost, and fallback ratio.

Applicable Scenarios

This approach is suitable for teams that:

1. Have already integrated two or more model providers.
2. Are migrating from Chat Completions to Responses, Messages, or Gemini GenerateContent.
3. Need to retain migration capability between OpenAI, Claude, Gemini, Bedrock, Vertex AI, and local models.
4. Have requirements for Agents, tool calling, structured output, streaming responses, content safety, or cost tracking.
5. Cannot tolerate JSON parsing failures, tool call anomalies, or billing inaccuracies due to model switching.

For personal projects or simple chat applications, using a unified SDK is fine. But once you enter a production system—especially one involving multi-tenancy, auditing, SLAs, cost accounting, and model replacement—you should build the adapter contract as early as possible.

References

1. OpenAI API Reference - Responses API
2. Anthropic Claude Docs - Using the Messages API
3. Anthropic Claude Docs - Tool use with Claude
4. Google AI for Developers - Gemini GenerateContent API
5. LiteLLM Documentation - Getting Started