LLM Multi-Tenant Quota Gateway in Production: Token Budgets, Rate Limits, and Fair Scheduling for Shared Model Pools

Background: The Biggest Fear for a Shared Model Pool is “Normal-Looking” Large Requests

When building an internal LLM platform, teams often focus first on model deployment, GPU utilization, inference throughput, prompt templates, and API compatibility. The real trouble emerges once multiple teams start using the platform: one tenant runs a batch summarization task, another maxes out the context window, another retries endlessly after failures, and another holds streaming connections open for too long.

These requests look normal at the HTTP layer, but their impact on LLM serving varies dramatically. A short Q&A might consume only a few hundred tokens, while contract review, customer service session compression, code analysis, or long document extraction can consume tens of thousands of tokens in a single request. More complex still, input tokens, output tokens, concurrent connections, queue wait times, KV cache usage, and model instance scheduling all affect both cost and latency simultaneously.

Therefore, a multi-tenant LLM platform cannot rely on simple “requests per second” rate limiting. It needs an LLM quota gateway that aligns more closely with inference costs: before a request enters the model pool, the gateway must handle tenant identification, budget checks, token estimation, rate limiting, priority assignment, queue admission, and cost attribution.

Core Principles: Quota is Not a Threshold, But a Set of Budget Ledgers

Layer 1: Identity and Tenant Context

When the gateway receives a request, it must first determine which tenant, application, environment, and cost center it belongs to. Don’t just rely on the API Key string itself; parse it into a structured identity:

{
  "tenant_id": "team-risk-control",
  "app_id": "contract-reviewer",
  "env": "prod",
  "plan": "gold",
  "cost_center": "legal-ai",
  "priority": "interactive"
}

This identity must flow through the entire pipeline: rate limiting, scheduling, logging, billing, alerting, and auditing should all use the same tenant context. If you only identify the tenant at the authentication stage, and the subsequent queue, model pool, and logging systems don’t carry the tenant field, you’ll never be able to answer “who filled up the GPU.”

Layer 2: Multi-Dimensional Quotas

LLM quotas should be divided into at least four categories:

Quota Type	Typical Metrics	Purpose
Request Rate Quota	RPM, RPS, per-user short window	Control request frequency, prevent burst spikes
Token Budget Quota	Input/output tokens per minute, total tokens per day, per-request context limit	Align with LLM cost, reflect actual resource consumption
Concurrency & Queue Quota	Max inflight requests per tenant, max queued requests, max streaming connections	Prevent one tenant from occupying all replicas’ ongoing requests
Cost Budget Quota	Daily USD budget, monthly budget, per-model budget, per-line-of-business budget	Prevent offline tasks and anomalous calls from blowing up the bill

Request Rate Quota: Envoy’s rate limit filter is a typical pattern. Configuration on a route or virtual host generates descriptors sent to the rate limit service; if a descriptor is exceeded, a 429 is returned. Descriptors can also be composed from multiple actions, which is suitable for expressing multi-dimensional rate-limiting keys like tenant, app, model, and route.

Token Budget Quota: More aligned with LLM cost than RPS, because the resource consumption difference between requests can span orders of magnitude.

Concurrency & Queue Quota: Ray Serve’s autoscaling documentation uses target_ongoing_requests and max_ongoing_requests as key parameters for defining system steady state and queue limits. This same idea applies at the tenant level: don’t let one tenant fill all replicas’ ongoing requests.

Cost Budget Quota: Not the sole basis for real-time performance protection, but it prevents offline tasks and anomalous calls from blowing up the bill.

Layer 3: Admission Control and Fair Scheduling

Quota checking is not a simple “pass or reject.” Production systems typically need four types of decisions:

type AdmissionDecision =
  | { action: "allow"; priority: number; budgetSnapshot: BudgetSnapshot }
  | { action: "queue"; queue: "interactive" | "batch"; ttlMs: number }
  | { action: "degrade"; maxOutputTokens: number; model: string; reason: string }
  | { action: "reject"; status: 429 | 402 | 503; retryAfterMs?: number };

Interactive requests can retain higher priority, batch tasks can be queued in a low-priority queue; requests that exceed the daily budget but are critical business can be degraded to a smaller model or have their output length shortened; clearly anomalous retry storms should be rejected outright with a clear Retry-After header.

NVIDIA Triton’s rate limiter documentation emphasizes that the rate limiter can control request scheduling across models and determine when model instances execute based on resources and priority. This shows that tenant-level fairness at the gateway is only the first step; the model serving layer itself also needs to know which requests should execute first and which model instances should be constrained by resources.

Engineering Implementation: From “Rate Limiter” to “Control Plane + Data Plane”

Data Plane: Every Request Must Be Accounted For Before Being Allowed

Once an LLM request enters the data plane, it can be processed in the following order:

1. Parse the API Key or JWT to obtain the tenant context.
2. Estimate input tokens and read request-declared parameters like max_tokens, model, stream, and tool_choice.
3. Query short-window quotas: RPS, RPM, TPM, concurrency, queue length.
4. Query long-window budgets: daily budget, monthly budget, model-level budget, cost center budget.
5. Generate an admission decision: allow, queue, degrade, or reject.
6. After the request completes, write back the ledger using actual input tokens, output tokens, latency, status code, and model price.

A simplified TypeScript pseudocode example:

async function admitLLMRequest(req: LLMRequest): Promise<AdmissionDecision> {
  const identity = await resolveTenantIdentity(req.apiKey);
  const estimatedInputTokens = estimateTokens(req.messages);
  const quotaKey = {
    tenantId: identity.tenantId,
    appId: identity.appId,
    model: req.model,
    priority: identity.priority,
  };

  const quota = await quotaStore.snapshot(quotaKey);

  if (quota.inflight >= quota.maxInflight) {
    return {
      action: "queue",
      queue: identity.priority === "interactive" ? "interactive" : "batch",
      ttlMs: 30_000,
    };
  }

  if (
    quota.tokensPerMinute.used + estimatedInputTokens >
    quota.tokensPerMinute.limit
  ) {
    return {
      action: "reject",
      status: 429,
      retryAfterMs: quota.tokensPerMinute.resetInMs,
    };
  }

  if (quota.dailyCost.usedUsd > quota.dailyCost.softLimitUsd) {
    return {
      action: "degrade",
      model: "small-fast-model",
      maxOutputTokens: Math.min(req.maxTokens ?? 1024, 512),
      reason: "daily_cost_soft_limit_exceeded",
    };
  }

  return {
    action: "allow",
    priority: priorityToWeight(identity.priority),
    budgetSnapshot: quota,
  };
}

The key point here isn’t the code itself, but the order: estimate first, then admit, finally settle based on actual consumption. Only accounting after the request completes cannot prevent large requests from entering the queue; only accounting before the request enters can lead to ledger drift due to inaccurate output token estimation. Both must be combined.

Control Plane: Quota Configuration Must Be Deployable, Rollbackable, and Auditable

Quotas should not be hardcoded. The control plane should at least support these configurations:

tenants:
  team-risk-control:
    plan: gold
    models:
      gpt-large:
        rpm: 120
        input_tpm: 200000
        output_tpm: 80000
        max_context_tokens: 32000
        max_output_tokens: 4096
        max_inflight: 24
        daily_budget_usd: 300
      gpt-small:
        rpm: 600
        input_tpm: 500000
        output_tpm: 200000
        max_inflight: 80
queues:
  interactive:
    weight: 8
    max_wait_ms: 3000
  batch:
    weight: 2
    max_wait_ms: 60000

These configurations need to go through approval, canary testing, and rollback. Especially when temporarily raising a tenant’s quota, you must record the reason, operator, effective time, expiration time, and scope of impact. Otherwise, temporary configurations become permanent risks.

Scheduling Layer: Avoid “Rich Tenants” Starving Others

Multi-tenant fair scheduling isn’t about making all tenants completely equal; it’s about allocating shared resources in an explainable way based on weights. Three common approaches are:

Scheduling Strategy	Principle	Use Case
Weighted Queues	Each tenant or tier has a weight; the scheduler pulls requests from queues based on weight	Simple and explainable, suitable for scenarios with clear tenant tiers
Token-aware Scheduling	Converts estimated input/output tokens and model type into work units	Scenarios with high request complexity variance
Priority + Aging	High-priority requests execute first; low-priority requests gradually increase in priority after waiting too long	Mixed online/offline task scenarios

Recent LLM serving scheduling research also incorporates client priority, SLOs, and request value into scheduling objectives, rather than just using FCFS.

Applicable Scenarios

• Internal Enterprise Model Platforms: Multiple business teams share the same set of model services and GPU pools. Each team has different calling patterns, so budgets and concurrency must be isolated.
• External SaaS Products: Free, standard, and enterprise tier users share the same LLM capabilities. Without tenant-level quotas, lower-priced plans can degrade the service quality of higher-priced plans.
• Agent Platforms: Agents often loop through model calls, tool calls, and retrieval systems, and may automatically retry on failure. A quota gateway must limit the maximum number of steps, maximum tokens, maximum tool calls, and maximum duration.
• Platforms with Mixed Online and Offline Tasks: Interactive tasks are latency-sensitive; offline tasks prioritize cost and throughput. They cannot share the same non-priority queue.

Common Pitfalls

Pitfall 1: Rate-Limiting Only by RPS

RPS only describes request count, not token consumption, output length, context window, streaming connection duration, or GPU occupancy. For LLM platforms, RPS must be used together with TPM, concurrency, queue length, and cost budgets.

Pitfall 2: Placing Rate Limiting Behind the Model Service

If requests have already entered the inference queue, the protective effect of rate limiting is greatly diminished. The correct position is before the model pool, immediately after receiving the request, parsing the identity, and estimating tokens.

Pitfall 3: Only Rejecting, Never Degrading

A quota gateway shouldn’t only return 429. For low-risk requests, you can shorten max_tokens, switch to a smaller model, disable expensive tools, move to an async queue, or prompt the user to check results later. Rejection should be the last resort.

Pitfall 4: Not Logging Rejection Reasons

Without rejection reasons, business teams will only see “the model isn’t working well.” Every rate limit, queue, and degradation should log a reason code, such as tenant_tpm_exceeded, daily_budget_soft_limit, model_pool_overloaded, or queue_timeout.

Go-Live Checklist

1. Can every request resolve tenant_id, app_id, model, priority, and cost_center?
2. Do you have RPS, TPM, concurrency, queue length, per-request limits, and cost budgets in place?
3. Are input tokens estimated before admission, and output tokens settled based on actual values after completion?
4. Do 429, 402, 503, degrade, and queue actions have clear reason codes and observable metrics?
5. Do you have dashboards at the tenant, model, queue, and global levels?
6. Does quota configuration support approval, canary testing, expiration, and rollback?
7. Can high-priority tenants infinitely crowd out low-priority tenants? Do low-priority tenants have an aging mechanism to prevent starvation?
8. Are batch tasks isolated from interactive tasks in separate queues?
9. Can you replay the last hour’s request ledger per tenant to explain billing and rate-limiting reasons?
10. Have you performed load testing with: short request storms, long-context requests, streaming requests, failure retry storms, single-tenant spikes, and multi-tenant simultaneous spikes?

FAQ

Q1: Should the token budget be based on input or output tokens?

Both. Input tokens determine context processing cost and some VRAM pressure; output tokens determine decode time, streaming connection duration, and the final bill. Output tokens can only be estimated before admission, so it’s recommended to pre-deduct based on max_tokens or the historical P95 output length, then correct with the actual output after completion.

Q2: Should a tenant be immediately disabled after exceeding their budget?

It’s not recommended to take a one-size-fits-all approach. You can use soft limits and hard limits. Soft limits trigger degradation, alerts, or moving to async; hard limits trigger rejection. Critical business tenants can also have emergency quotas, but these must come with approval and an expiration time.

Q3: Do fair scheduling and priority scheduling conflict?

No. Fair scheduling ensures long-term resource allocation is explainable; priority scheduling ensures critical requests are processed first in the short term. Production systems typically use weighted fairness: high-priority tenants have a higher weight, but low-priority tenants are not permanently starved.

References

• Envoy Rate Limit Filter
• NVIDIA Triton Inference Server Rate Limiter
• Ray Serve Advanced Autoscaling
• PROSERVE: Unified Multi-Priority Request Scheduling for LLM Serving