LLM Serving Overload Protection: SLO-Driven Autoscaling and Admission Control

The Core Problem: LLM Services Fear Loss of Control More Than Slowness

After deploying a large language model inference service, teams often focus on optimizing single-request latency, throughput, or GPU utilization. However, in a real production environment, the most critical issue is rarely “a single slow request,” but rather the system entering an unrecoverable queuing state under a traffic spike.

Typical symptoms include:

• The request entry point continues to accept traffic, but the backend GPU queue is already piling up.
• P50 latency looks acceptable, but P95/P99 latency is rapidly deteriorating.
• Autoscaling has been triggered, but new replicas’ cold starts, weight loading, and model initialization can’t keep up with the traffic peak.
• Client retries amplify the traffic, creating a retry storm.
• The result isn’t a few request failures, but a massive, simultaneous timeout of many requests.

Therefore, production governance for LLM Serving cannot just ask “how to increase throughput.” It must also ask: When capacity is insufficient, how should the system degrade, queue, scale, reject, and recover in a controlled manner?

The core idea of this article is to build an overload protection loop for LLM inference services using SLO-driven autoscaling and admission control.

Core Principles: Shifting from Resource Metrics to SLO and Token Pressure

Traditional web services often use CPU, memory, QPS, and average latency for scaling decisions. However, LLM inference has several unique characteristics that make these metrics insufficient.

Highly Non-Uniform Request Costs

Two requests might both have a QPS of 1, but their costs can be completely different. One request might have only a few dozen input and output tokens; another might have tens of thousands of input tokens and require a long generated answer. For LLM Serving, the real resource consumers are token workload, KV Cache, GPU memory usage, decoding steps, and batching scheduling space.

Therefore, admission control cannot be based solely on request count. At a minimum, requests should be broken down by:

Dimension	Impact
input_tokens	Affects prefill pressure and TTFT
max_output_tokens	Affects decode duration and TPOT
tenant_id	Affects quotas, fairness, and cost attribution
model_id	Affects weight loading, memory pool, and instance selection
priority	Distinguishes interactive, background, and batch tasks

GPU Utilization is a Lagging Signal

High GPU utilization doesn’t necessarily mean the system is healthy, and low utilization doesn’t mean it’s idle. In LLM inference, queuing can occur at the entry point, scheduler, KV Cache allocation, network transfer, model replica cold start, and other locations. Relying solely on GPU utilization often means you only know there’s a problem after it has already occurred.

Ray Serve’s official documentation also bases autoscaling on request queues and ongoing requests, rather than just underlying resource utilization. target_ongoing_requests indicates the average number of in-flight requests the autoscaler aims to maintain per replica. This value should be tuned based on request processing time and latency targets.

SLO is the Boundary for Protection Strategies

Common SLOs for LLM services can be broken down into three categories:

Metric	Meaning	Protection Focus
TTFT	Time To First Token	Whether the user feels the system is stuck
TPOT / TBT	Interval between output tokens	Whether streaming output is smooth
E2E Latency	End-to-end completion time	Experience for long answers, batch, and background tasks

If a request entering the system will inevitably exceed its SLO, continuing to accept it isn’t necessarily better. A better approach is to quickly assess at the entry point: serve if possible; otherwise, degrade, queue, return a 429/503, or suggest the client retry later.

Engineering Implementation: Building a Five-Layer Overload Protection Loop

Layer 1: Ingress Rate Limiting, Controlling Traffic Inflow First

The goal of ingress rate limiting isn’t simply to reject requests, but to prevent the system from exceeding a recoverable load level. It’s recommended to implement rate limiting based on at least the following dimensions:

• Global QPS limit.
• Per-tenant QPS limit.
• Per-tenant Token/min limit.
• Per-model concurrency limit.
• Per-request max input and output token limits.
• Stricter limits for high-risk interfaces, such as agents, tool calling, and long-context analysis.

A practical rule is: Upgrade the rate limiting unit from requests to token budget. For example:

tenant_policy:
  tenant_a:
    requests_per_minute: 600
    input_tokens_per_minute: 2_000_000
    output_tokens_per_minute: 800_000
    max_concurrent_requests: 80
    max_input_tokens_per_request: 32_000
    max_output_tokens_per_request: 2_000

This prevents a single tenant from exhausting the entire model pool with a few very long requests.

Layer 2: Admission Control, Determining if a Request is Worth Queuing

Admission control is closer to the inference service itself than rate limiting. It considers not only tenant quotas but also the current service state:

• Current queue length.
• Estimated wait time.
• Available KV Cache or GPU memory budget.
• Current number of model pool replicas.
• Number of replicas in cold start.
• Historical TTFT/TPOT SLO attainment rate.
• Request’s remaining SLO budget.

A simplified decision logic can be used:

def admit(request, state):
    estimated_tokens = request.input_tokens + request.max_output_tokens
    estimated_wait_ms = state.queue_delay_ms_p95
    estimated_service_ms = estimate_service_time(request, state)
    predicted_latency = estimated_wait_ms + estimated_service_ms

    if request.tenant.remaining_token_budget < estimated_tokens:
        return "reject_quota"
    if state.kv_cache_free_blocks < estimate_kv_blocks(request):
        return "reject_memory_pressure"
    if predicted_latency > request.slo_ms:
        if request.priority == "background":
            return "defer"
        return "reject_fast"

    return "admit"

The key here isn’t absolute estimation accuracy, but avoiding piling up requests when it’s clear the SLO cannot be met.

Layer 3: Queue Prioritization, Preventing High-Priority Requests from Being Stalled by Long Tasks

A production system should not have a single FIFO queue. At a minimum, it should distinguish:

• Interactive requests: Chat, search, customer service Q&A, sensitive to TTFT.
• Background requests: Batch summarization, offline analysis, low-priority automated tasks.
• High-value tenant requests: Higher paid tier, stricter SLOs.
• Long-context requests: High token cost, can easily slow down the queue.

In multi-tenant scenarios, you also need to prevent one tenant from consuming all capacity. Strategies like weighted fair queue, priority queue, token bucket, and deadline-aware queue can be used. Recent research on multi-tenant GPU inference also notes that output length and runtime token drift can affect scheduling accuracy, so the scheduler needs runtime feedback correction, not just estimates from when the request entered.

Layer 4: Autoscaling, Based on Queues and SLOs, Not Single-Point Resource Metrics

The input for autoscaling should include at least:

• Queue length and queuing time.
• Ongoing requests per replica.
• TTFT P95/P99.
• TPOT P95/P99.
• Input tokens per second, output tokens per second.
• SLO violation rate over the past few minutes.
• Number of replicas starting up.
• Cold start time distribution.

The Chiron paper proposes hierarchical backpressure based on queue size, utilization, and SLO for LLM Serving autoscaling. Its core insight is that scaling strategies should incorporate request type and SLO pressure, rather than treating all requests as identical QPS.

For LLM services, these mechanisms can be further extended:

autoscaling:
  min_replicas: 2
  max_replicas: 20
  target_ongoing_requests: 2
  max_ongoing_requests: 4
  scale_up_signal:
    - queue_delay_p95_ms > 800
    - ttft_p95_ms > 1500
    - input_tokens_per_second > baseline * 1.4
  scale_down_signal:
    - queue_delay_p95_ms < 200 for 10m
    - gpu_memory_pressure < 60% for 10m
    - slo_violation_rate < 1% for 10m

Avoid two common pitfalls:

1. Scaling up is not instantaneous. Large model replicas may require pulling images, loading weights, initializing the inference engine, and warming up CUDA kernels.
2. Scaling up doesn’t solve all bottlenecks. If the bottleneck is a shared gateway, KV Cache migration, tenant quotas, or downstream tool calls, simply adding replicas is ineffective.

Layer 5: Degradation and Fast Failure, Preventing Avalanche Propagation

Don’t just “wait” during overload. Optional strategies include:

• Return 429 for low-priority requests with a Retry-After header.
• Defer background tasks to a delayed queue.
• Shorten the maximum output tokens.
• Switch to a smaller model.
• Disable high-cost features like multi-turn tool calling, deep reasoning, or long-context expansion.
• Implement idempotent caching or result reuse for identical requests.
• Set retry budgets for client retries to prevent infinite retries.

A mature system should clearly distinguish between these states:

State	System Behavior
Normal	Accept requests, schedule by priority
Mild Congestion	Reduce low-priority concurrency, increase scaling sensitivity
Moderate Congestion	Rate-limit some tenants, defer background tasks
Severe Congestion	Fast failure, protect core interactive requests
Recovery	Gradually relax rate limiting, prevent sudden traffic surge

Applicable Scenarios

This mechanism is particularly suitable for:

1. Conversational products: Users are sensitive to first-token latency, requiring TTFT protection.
2. Enterprise multi-tenant platforms: Different customers have different quotas, priorities, and cost attribution.
3. Agent platforms: Requests may trigger tool calls, long-chain reasoning, and multiple model invocations, leading to high cost variability.
4. Internal AI gateways: Multiple services share the same model pool, requiring quotas and fairness.
5. Serverless GPU inference: Cold starts and weight loading significantly impact scaling speed.

Common Misconceptions

Misconception 1: High GPU Utilization is Always Good

High GPU utilization might mean resources are fully used, or it might mean the system has no scheduling headroom. LLM Serving should focus on goodput, i.e., effective throughput that meets SLOs, not just raw throughput.

Misconception 2: All Requests Should Queue and Wait

If a request will inevitably time out after waiting, queuing only consumes resources and amplifies tail latency. In this case, fast failure is better for user experience and system recovery.

Misconception 3: Autoscaling Can Replace Rate Limiting

Autoscaling has latency and is affected by GPU inventory, model loading, schedulers, and network constraints. Ingress rate limiting, admission control, and autoscaling must work together.

Misconception 4: Billing and Rate Limiting by Request Count is Sufficient

The true cost of an LLM request is determined by tokens, model, context length, output length, and inference strategy. Rate limiting by request count underestimates the pressure from long-context and long-output requests.

Production Readiness Checklist

Metrics Layer

• Are TTFT, TPOT, and E2E latency P50/P95/P99 collected?
• Are input tokens, output tokens, and tokens/sec differentiated?
• Can queue delay, running requests, and waiting requests be seen?
• Can metrics be broken down by tenant, model, and priority?
• Are the reasons for rejected, deferred, and degraded requests recorded?

Strategy Layer

• Is there rate limiting at the global, tenant, and model levels?
• Is there a token budget, not just QPS?
• Are priorities defined for interactive, background, and batch requests?
• Is there a fast-failure strategy and Retry-After?
• Is there a retry budget to prevent client traffic amplification?

Scaling Layer

• Has the model replica cold start time been measured?
• Are min_replicas set to avoid complete cold starts?
• Are queue and SLO metrics used to trigger scaling?
• Is max_replicas set to prevent cost overruns from unlimited scaling?
• Are starting, ready, and unavailable replicas distinguished?

Drill Layer

• Has traffic spike stress testing been performed?
• Has stress testing with mixed long-context requests been performed?
• Has malicious or abnormally high load testing for a single tenant been performed?
• Has the rate limiting recovery process been validated?
• Has it been verified that monitoring alerts can pinpoint the specific tenant and model?

References

• Ray Serve Autoscaling Guide: https://docs.ray.io/en/latest/serve/autoscaling-guide.html
• Chiron: Hierarchical Autoscaling for Large Language Model Serving: https://arxiv.org/abs/2501.08090
• AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure: https://arxiv.org/abs/2504.03648
• TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity: https://arxiv.org/abs/2512.03416
• DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference: https://arxiv.org/abs/2606.02982