LLM Gateway in Production: Governing Multi-Tenancy, Budgets, and Model Access with a Policy Engine

The Problem: Model Calls Are Evolving from “API Integration” to “Governance Entry Point”

Many teams initially integrate large language models by simply dropping API keys from OpenAI, Anthropic, Google, local models, or cloud providers into their application configuration. This phase seems simple, but as the business moves from a single demo to a production system, problems quickly surface:

• Different teams, tenants, and applications share the same set of keys, making cost tracking impossible.
• A test script or anomalous user can suddenly consume a massive number of tokens.
• Free users, paid users, internal employees, and automated agent tasks all use the same model permissions.
• Model upgrades, prompt version changes, and Provider switches lack canary and rollback mechanisms.
• When an issue occurs in production, only application logs are available, not request-level details like tokens, model, cost, error codes, and retry chains.

These problems cannot be solved long-term by “adding a few more if statements.” As model calls become an infrastructure capability, teams need a unified entry point: an LLM Gateway.

The value of an LLM Gateway is not simply forwarding requests; it’s transforming model calls into governable, observable, rate-limited, auditable, and releasable production traffic.

Core Principle: The LLM Gateway is the Policy Enforcement Point for Model Calls

A production-grade LLM Gateway must address at least five categories of problems:

1. Identity: Which tenant, user, application, environment, and business scenario does the current request belong to?
2. Access Control: Is this identity allowed to call the specified model, use specific tools, use a specific context length, or use a specific output mode?
3. Budget & Rate Limiting: Has the request exceeded RPM, TPM, concurrency, daily budget, monthly budget, or project budget limits?
4. Request Governance: Does the request require retry, degradation, caching, Provider fallback, model alias mapping, or canary release?
5. Audit & Observability: Log the request, tokens, cost, latency, errors, policy hits, and version information.

Cloudflare AI Gateway positions itself as the entry point to “observe and control AI applications,” with features covering analytics, logging, caching, rate limiting, request retry, and model fallback. LiteLLM Proxy uses virtual keys to track spend and control model access, while supporting user, team, and key-level budgets, RPM, TPM, and multi-window budgets. Kong AI Gateway emphasizes provider-agnostic APIs, centralized credential management, access tiers, usage analytics, rate limiting, semantic cache, guardrails, and prompt engineering. OpenAI’s official rate limit documentation also clarifies that production API limits include not just RPM, but also RPD, TPM, TPD, and organization/project-level constraints.

From these products and documentation, it’s clear that the core of an LLM Gateway is not a “unified URL,” but a unified policy boundary.

Recommended Architecture: Separation of Control Plane, Data Plane, and Policy Plane

In a production environment, it’s not advisable to cram all logic into a single reverse proxy. A more robust architecture involves a three-layer separation:

Data Plane: Low-Latency Forwarding and Basic Protection

The data plane handles real-time requests:

• Parse API keys, JWTs, and tenant headers.
• Normalize the request into an OpenAI-compatible format.
• Enforce basic rate limiting, timeouts, retries, and circuit breaking.
• Call the policy service for admission decisions.
• Forward requests to the target model Provider or internal inference service.
• Write request logs, token estimates, response summaries, and error information.

The data plane must be stable, low-latency, and horizontally scalable. It should not handle complex approval workflows or dynamically load too many business rules.

Control Plane: Configuration, Versioning, Release, and Rollback

The control plane manages configuration:

• Tenants, applications, user groups, environments, and API keys.
• Model whitelists, model aliases, default models, and fallback models.
• Budgets, rate limits, concurrency limits, context lengths, and maximum output tokens.
• Prompt template versions, canary percentages, release statuses, and rollback records.
• Audit queries, cost reports, and anomaly alerts.

The control plane outputs publishable policy snapshots, rather than forcing the data plane to query multiple business tables for every request.

Policy Plane: An Independent Decision Engine

The policy plane answers a specific question: Is the current request allowed in the current context? If so, which model, budget window, degradation rules, and audit tags should be used?

The policy plane can be custom-built or can follow the approach of OPA / Envoy external authorization. The OPA-Envoy documentation explains that Envoy can pass request context to an external authorization service for context-aware access control. OPA also supports dry-run mode, which is useful for observing “what would have been blocked” during policy refactoring or initial integration. This is highly insightful for an LLM Gateway: new policies should not be directly enforced on production traffic; they should first be shadowed / dry-run, then gradually enforced.

Policy Model: Don’t Just Manage Keys; Manage “Tenant + Scenario + Model + Cost”

Many teams simplify their LLM Gateway to just API key management, which is insufficient. It’s recommended to establish at least the following policy dimensions.

Tenant Dimension

Every request must be tied to a stable tenant_id. Don’t rely solely on user emails or application names, as these fields can change easily.

tenant:
  id: tenant_enterprise_a
  plan: enterprise
  environments: [prod, staging]
  monthly_budget_usd: 5000
  default_model_group: premium-chat

Application Dimension

Within the same tenant, the risk and budget profile of a customer service agent, a data analysis assistant, a code assistant, and a backend batch job are completely different.

application:
  id: support_agent
  tenant_id: tenant_enterprise_a
  allowed_models: [gpt-5-mini, claude-sonnet, internal-qwen]
  max_context_tokens: 64000
  max_output_tokens: 4096
  allow_streaming: true

Scenario Dimension

Even within the same application, different scenarios need to be distinguished. For example, “user online Q&A” and “nightly batch summarization” should not share the same SLO and budget.

scenario:
  id: realtime_support_reply
  rpm_limit: 120
  tpm_limit: 600000
  max_parallel_requests: 40
  retry_budget: 1
  fallback_model_group: standard-chat

Model Access Dimension

Model permissions should not be defined only on the client side. The gateway needs to centrally enforce:

• Can this tenant use high-cost models?
• Is long context allowed?
• Is multimodal input allowed?
• Are reasoning models or tool calls allowed?
• Is access to a specific privately deployed model allowed?

Enforcing these checks on the client side provides no security; they must be enforced server-side.

A Practical Request Decision Flow

An LLM Gateway request can be processed using the following flow:

Request -> Authenticate identity -> Normalize provider-compatible payload
-> Resolve tenant/application/scenario -> Load policy snapshot -> Check
model access -> Check RPM/TPM/concurrency/budget -> Apply prompt/model
version -> Route to provider or internal serving -> Record usage, cost,
latency, policy decision -> Return response

The most critical step is the policy snapshot. Do not assemble rules on the fly for each request, as this makes it impossible to reproduce production behavior. Each policy release should generate an immutable version:

policy_snapshot:
  version: gateway-policy-2026-07-01-001
  status: active
  created_by: platform-team
  effective_at: 2026-07-01T10:00:00Z
  rules:
    - id: enterprise-premium-model-access
    - id: free-tier-token-budget
    - id: batch-job-night-window

The snapshot version must be recorded in the request log. Otherwise, when a user reports “it worked yesterday, but not today,” the platform team cannot determine if the cause is a model change, budget exhaustion, rate limiting, or a policy release.

Engineering Implementation: Start with Three Rule Types, Don’t Try to Build a Giant Platform

It’s easy for an LLM Gateway to become a massive governance platform. A more practical approach is to first implement three types of rules.

Type 1: Access Rules

Access rules answer “can it be used?” Typical fields include:

Field	Description
tenant_id	Tenant identifier
app_id	Application identifier
environment	Environment (prod/staging/dev)
allowed_models	List of allowed models
denied_models	List of denied models
max_context_tokens	Maximum context length
max_output_tokens	Maximum output length
allow_tools	Whether tool calls are allowed
allow_multimodal	Whether multimodal input is allowed

Example rule (OPA style):

package llm.gateway.authz

default allow := false

allow if {
    input.tenant.plan == "enterprise"
    input.request.model in input.policy.allowed_models
    input.request.context_tokens <= input.policy.max_context_tokens
}

These rules should be as deterministic as possible. Do not rely on the model itself to determine if a model call is allowed. Access control must be explainable and reproducible.

Type 2: Budget Rules

Budget rules answer “can it still spend?” At least four layers are needed:

Layer	Purpose	Example
Key-level budget	Prevent a single key from leaking or being abused	Daily budget $50
User-level budget	Prevent a single user from anomalous consumption	Daily budget $10
Team / Tenant-level budget	Prevent an entire customer or team from overspending	Monthly budget $5000
Model-level budget	Prevent high-cost models from being misused by low-value scenarios	Daily budget $200

LiteLLM’s multi-window budget_limits is a great reference: the same key can have both a daily and monthly budget. In production, it’s recommended to add an hourly soft threshold for early warning, rather than waiting until the monthly budget is exhausted to block requests.

Type 3: Rate and Concurrency Rules

Rate rules answer “can it enter now?” Don’t just configure RPM. The cost of an LLM request is highly correlated with the number of tokens, context length, and output length. A more reasonable rate limiting strategy should combine the following dimensions:

• RPM (Requests Per Minute): Request count limit.
• TPM (Tokens Per Minute): Token throughput limit.
• max_parallel_requests: Concurrent request limit.
• max_queued_requests: Queue size limit.
• max_estimated_cost: Maximum estimated cost per request.

OpenAI’s official documentation mentions that rate limits use metrics like RPM, RPD, TPM, and TPD, and hitting any one of these dimensions can trigger a limit. For a custom Gateway, multi-dimensional rate limiting should also be used, not just request count limiting.

Release Governance: Policies Must Support Dry-Run, Canary, and Rollback

Changes to an LLM Gateway’s policies are a production risk. An incorrect rule can lead to:

• A large number of users being incorrectly blocked.
• Free users accessing high-cost models.
• Batch processing tasks bypassing budgets.
• A specific tenant suddenly being unable to call core capabilities.
• A wrong prompt version causing a drop in output quality.

It is recommended that each policy release includes four stages:

1. lint: Check fields, model names, budget units, time windows, and reference relationships.
2. dry-run: Only record would_allow / would_deny, do not actually block.
3. canary: Apply to a small number of tenants, applications, or fixed test keys.
4. enforce: Apply to all traffic, with a one-click rollback capability.

OPA-Envoy’s dry-run design is highly relevant here: observe the decision logs of a new policy before deciding whether to enforce it. An LLM Gateway should also log policy_decision, reason_code, and matched_rule_id.

Example audit log:

{
  "request_id": "req_20260701_001",
  "tenant_id": "tenant_enterprise_a",
  "app_id": "support_agent",
  "model": "gpt-5-mini",
  "policy_snapshot": "gateway-policy-2026-07-01-001",
  "decision": "allow",
  "matched_rules": ["enterprise-premium-model-access"],
  "estimated_input_tokens": 1830,
  "estimated_output_tokens": 512,
  "cost_center": "customer-support",
  "latency_ms": 1320
}

Common Pitfalls

Pitfall 1: Treating the Gateway as a Provider SDK Wrapper

An SDK wrapper only reduces integration costs; it cannot inherently solve budget, access control, audit, and release governance problems. The Gateway must be a server-side enforcement point.

Pitfall 2: Only Doing Model Routing, Not Permission Boundaries

Model routing can optimize cost and availability, but without tenant permissions, budgets, and audits, the more flexible the routing, the greater the risk.

Pitfall 3: Budgets Only Tracked Monthly

Monthly budgets are suitable for financial management, but not for preventing incidents. In production, you need minute-level, hourly, daily, and monthly windows.

Pitfall 4: Policy Changes Without Version Numbers

Without version numbers, you cannot perform a post-mortem. Every request must record the policy version, model version, prompt version, and key rate-limiting results.

Pitfall 5: Putting All Rules in Business Code

Business code can handle business semantics, but platform-level model access, budgets, and audits should be centrally governed. Otherwise, multiple services will experience rule drift.

Go-Live Checklist

Identity and Tenancy

• Can every request be mapped to a tenant_id, app_id, and environment?
• Are anonymous requests blocked from directly accessing the model Provider?
• If an API key is leaked, can it be quickly disabled and its historical usage traced?

Policy and Release

• Do policies have version numbers and a state machine?
• Is dry-run, canary, enforce, and rollback supported?
• Is there policy linting and conflict checking?
• Are matched_rule_id and reason_code logged?

Budget and Rate Limiting

• Are RPM, TPM, concurrency, and budget windows all supported?
• Are key, user, team, tenant, and model-level budgets differentiated?
• Is there a runaway usage alert?
• Are failed retries also counted towards rate limits and budget statistics?

Observability and Audit

• Are model, Provider, tokens, cost, latency, and error codes logged?
• Can costs be aggregated by tenant, application, scenario, and model?
• Can the prompt version and policy version used for a specific request be identified?
• Can audit logs be exported for security and finance teams?

Reliability

• Is there a fallback if a Provider times out?
• Does the fallback model still meet business compliance requirements?
• Does the Gateway itself have circuit breaking and rate limiting?
• When the policy service is unavailable, is the behavior fail-open or fail-closed? Is this differentiated by scenario?

Applicable Scenarios

An LLM Gateway is particularly suitable for the following scenarios:

• Multiple business teams sharing model resources.
• Multi-tenant SaaS products that need to bill or limit customers.
• Using multiple model Providers simultaneously.
• Having different permission tiers like Free, Pro, and Enterprise.
• Handling mixed traffic from Agents, batch processing, online customer service, and internal assistants.
• Needing to allocate model costs to departments, customers, or projects.
• Requiring canary governance for prompts, models, and policy releases.

If you only have a single internal script that occasionally calls a model, you probably don’t need a full Gateway. But once you enter production with multiple teams, tenants, models, and budget boundaries, a Gateway becomes infrastructure.

References

1. LiteLLM Virtual Keys Documentation
2. LiteLLM Budgets and Rate Limits Documentation
3. Cloudflare AI Gateway Documentation
4. Cloudflare AI Gateway Rate Limiting Documentation
5. Kong AI Gateway Documentation
6. OpenAI Rate Limits Documentation
7. OPA-Envoy Plugin Documentation