Does backpressure reduce system throughput?

Properly configured backpressure is not about proactively reducing throughput—it prevents the queue from growing unboundedly. Returning 429 or 503 once the system exceeds recoverable capacity is typically more stable than letting requests sit in a long queue and eventually time out.

LLM Serving Overload Protection in Production: Guarding Tail Latency with Queue SLO, Backpressure, and Elastic Autoscaling

Q: Why can't LLM Serving autoscaling rely on GPU utilization alone?

GPU utilization is a lagging indicator—it cannot express how long requests have been queued or how much TTFT budget remains. Production systems should jointly monitor waiting requests, queue time, TTFT, and replica cold-start time before deciding to scale or throttle.

Q: How should the autoscaler's target concurrency be determined?

Don't guess from experience. Run load tests with a fixed model, GPU, input/output length distribution, and inference parameters. Find the safe per-replica concurrency under your target TTFT/TPOT constraints, then set the autoscaler target below that boundary with headroom for bursts.

The most common failure in production-grade LLM inference services isn’t the GPU running out of memory—it’s a sudden traffic spike that saturates existing replicas while new requests keep piling into the queue. By the time the autoscaler detects pressure, requests a GPU, boots the container, and loads the model, the TTFT (Time To First Token) for those earlier requests has already blown past the SLO.

The root cause is a race against time: queue degradation almost always outpaces GPU replica scaling.

That’s why capacity management for LLM serving can’t rely on autoscaling alone. It needs three layers working together:

SLO awareness: Detect whether the queue is consuming the user’s acceptable wait budget.
Backpressure / Load Shedding: Stop unbounded queuing when the system can no longer absorb new requests within SLO.
Autoscaling: Increase long-term available throughput to close the supply-demand gap.

In short: autoscaling adds supply, backpressure caps excess demand, and the SLO tells the system when it must act.

Core Principles: From Queue Length to Queue SLO

1. Don’t Treat GPU Utilization as Your Only Scaling Signal

Traditional web services scale on CPU or memory utilization, but LLM inference has unique characteristics: request lengths vary wildly, Prefill and Decode have different load profiles, concurrent scheduling changes per-request latency, and high GPU utilization doesn’t necessarily mean the SLO is already broken.

More direct production signals typically include:

vllm:num_requests_running: number of requests currently executing;
vllm:num_requests_waiting: number of requests waiting to be scheduled;
vllm:request_queue_time_seconds: actual time requests spend in the queue;
vllm:time_to_first_token_seconds: TTFT;
vllm:e2e_request_latency_seconds: end-to-end latency.

vLLM’s production metrics already expose these dimensions directly, so your autoscaler doesn’t have to rely on GPU utilization alone.

2. Queue Length Is a Capacity Signal; Queue Time Is What Users Actually Feel

Suppose both scenarios have 20 waiting requests:

Model A can drain 10 requests per second;
Model B can drain only 1 request per second.

The queue length is identical, but the user-facing wait risk is completely different. A more sensible control logic should care about:

Is the estimated wait time + current queue time + new replica cold-start time still within the TTFT SLO budget?

You can abstract this into a simple decision:

remaining_slo_budget = ttft_slo - current_queue_time

if estimated_scale_up_time > remaining_slo_budget:
    start_backpressure()
else:
    scale_out()

This isn’t asking the production system to predict every request precisely—it establishes a fundamental principle: if scaling can’t save the current requests in time, don’t keep feeding them into an unbounded queue.

3. Autoscaling and Backpressure Must Coexist

Ray Serve provides two critical parameters with very different semantics:

max_ongoing_requests: limits how many requests each replica accepts concurrently;
max_queued_requests: limits how many requests the proxy or caller is allowed to keep queuing.

Once max_queued_requests is reached, you can trigger backpressure directly and return HTTP 503; via BackpressureConfig you can also configure 429 with a Retry-After header.

This is far better suited to online inference with strict SLOs than “accept everything first, then queue slowly”—because an unbounded queue just hides capacity shortfalls as tail latency problems.

Engineering Implementation: A Four-Layer Control Loop

Layer 1: Define Request-Level SLOs

Classify requests by business type—don’t use a single global timeout for everything. For example:

Request Type	Key Metrics	Recommended Governance
Online chat	TTFT P95 / P99, TPOT	Keep a warm pool, scale fast, strictly cap the queue
Agent tool calls	End-to-end latency, TTFT	Moderate queue, allow limited retries
Background generation	Completion time	Can use async queues, prioritize cost efficiency
Batch inference	Job completion time	Should not compete with interactive requests for the same SLO

The key isn’t the specific thresholds—it’s that interactive and batch workloads must use different capacity strategies.

Layer 2: Alert on Waiting, Queue Time, and TTFT

At minimum, set up the following metrics:

P95 / P99 TTFT
P95 queue_time
num_requests_waiting / num_requests_running
request_reject_rate
replica_ready_latency
GPU replica count

replica_ready_latency is easy to overlook. It should cover the full time from the autoscaler’s scale-out decision to when the GPU pod can actually accept requests—including scheduling, image startup, model weight loading, and initialization. If your model takes tens of seconds or even minutes to become ready, simply making the autoscaling observation window faster won’t solve the problem.

Layer 3: Autoscale on Sustained Per-Replica Pressure

Ray Serve’s autoscaler uses target_ongoing_requests as the desired average number of ongoing requests per replica, with parameters like upscale_delay_s, downscale_delay_s, min_replicas, and max_replicas.

KServe can use KEDA with Prometheus or OpenTelemetry metrics to drive scaling; its official example uses vLLM’s vllm:num_requests_running directly as the scaling metric.

In production, don’t rush into complex predictive models. A more reliable path is:

Load-test to find the safe per-replica concurrency under your target TTFT;
Set the autoscaler target below that safe concurrency;
Measure actual scale-up latency;
Use that latency to decide how many minReplicas or warm pool instances you need;
Then evaluate whether predictive scaling is worth adding based on historical traffic.

Layer 4: Actively Reject When Overloaded—Don’t Queue Indefinitely

Consider dividing system state into three zones:

State	Trigger	Action
GREEN	`queue_time < 40% TTFT budget`	Accept normally
YELLOW	`queue_time >= 40% TTFT budget` and waiting requests keep growing	Trigger fast scale-out, deprioritize non-critical traffic
RED	`queue_time + estimated_scale_up_time >= TTFT budget`	Backpressure / load shedding, return 429 or 503 + `Retry-After`

Thresholds should be calibrated through load testing, not copied mechanically. But a three-state machine is far easier to explain and operate than a naive “scale when concurrency exceeds X.”

An Emerging Direction Worth Noting: The Scaling Unit Doesn’t Always Have to Be a Full Model Replica

OpScale, published in August 2026, explores a finer-grained research direction: traditional autoscaling treats a full model replica as the smallest scaling unit, but different operators have different runtime resource elasticity. This opens the door to Operator-level Provisioning and Autoscaling.

This direction is better suited as frontier architecture research than as a production blueprint for most teams to copy directly. But it highlights an important fact: the cost and response speed of LLM inference scaling are heavily influenced by the granularity of scaling.

For most teams today, the more practical choice remains full-replica scaling + warm pool + backpressure. Only when your GPU cluster is large enough, SLOs are extremely strict, and resource costs are sensitive enough should you consider finer-grained elasticity mechanisms.

When to Use This Approach

This approach is especially well-suited to:

Public-facing Chat / Copilot services with highly variable traffic;
Shared internal LLM gateways;
Multi-tenant GPU inference platforms;
Intelligent customer service and search Q&A with strict TTFT requirements;
Kubernetes inference services built on KServe, Ray Serve, vLLM, and similar.

If your workload is offline batch processing where users don’t care about second-level TTFT, shift your priorities to GPU utilization and per-token cost instead of forcing online overload protection logic.

Common Pitfalls

Pitfall 1: “The GPU still has headroom, so the service isn’t overloaded”

Whether an LLM service is overloaded should be judged by whether the SLO is consistently met, not just GPU utilization. If the queue is growing rapidly, user experience may already be degrading even when GPU metrics haven’t hit 100%.

Pitfall 2: “If scaling is fast enough, we don’t need rate limiting”

Scaling has physical latency. GPU scheduling, image startup, and model loading can never be zero-time. With a large enough traffic spike, any autoscaler needs backpressure as a safety net.

Pitfall 3: “A bigger max queue is safer”

A large queue doesn’t create throughput—it just postpones failure from “immediate rejection” to “timeout after a long wait.” For interactive workloads, oversized queues usually mean worse P99 latency.

Pitfall 4: “minReplicas=0 is always the cheapest”

Scale-to-zero saves idle GPUs but introduces cold starts. If model weight loading time approaches or exceeds your user’s TTFT SLO, scale-to-zero fundamentally conflicts with interactive service goals.

Launch Checklist

Before going live, complete at least the following:

FAQ

Why can’t LLM Serving autoscaling rely on GPU utilization alone?

Because GPU utilization can’t directly express how long requests have been queued or how much TTFT budget remains. Production systems should at least combine waiting requests, queue time, TTFT, and replica startup latency.

Should backpressure return 429 or 503?

If the semantics are “this tenant or caller has sent too many requests, retry later,” 429 is easier for clients to understand. If the semantics are “the service is temporarily unavailable or under-provisioned overall,” 503 is more natural. Ray Serve’s backpressure configuration supports both status codes and can include Retry-After.

How should the autoscaler’s target concurrency be determined?

Don’t guess from experience. Fix the model, GPU, input/output length distribution, and inference parameters, then run load tests. Find the safe per-replica concurrency under your target TTFT/TPOT constraints, set the autoscaler target below that boundary, and leave headroom for bursts.

References

Ray Serve Autoscaling Guide — https://docs.ray.io/en/latest/serve/autoscaling-guide.html
Ray Serve Advanced Autoscaling — https://docs.ray.io/en/latest/serve/advanced-guides/advanced-autoscaling.html
Ray Serve Production Best Practices — Load Shedding — https://docs.ray.io/en/master/serve/production-guide/best-practices.html
vLLM Production Metrics — https://docs.vllm.ai/en/latest/usage/metrics/
KServe — Autoscale InferenceService with LLM Metrics — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
OpScale: Operator-level Provisioning and Autoscaling for LLM Serving — https://arxiv.org/abs/2608.13499