LLM Serving Autoscaling in Production: Safe Scaling with Queue Depth, P95 Latency, and GPU Utilization

The Problem: Why LLM Serving Can’t Use Standard Web Service Autoscaling

Traditional web service autoscaling relies on CPU, memory, QPS, or request concurrency. In the LLM Serving world, these metrics break down: a single request with a short prompt vs. a long prompt can have vastly different computational costs; at the same 10 QPS, differences in context length, output tokens, batching state, and model size create wildly different GPU pressure.

The most common production mistake is treating GPU utilization or CPU utilization as the sole autoscaling signal. This typically leads to three problems:

1. Low GPU utilization, but users already feel slowness. Requests pile up in the ingress queue or internal replica queues. P95 latency spikes, while GPU utilization still shows “headroom.”
2. Scaling actions have a delayed effect. Loading large model weights, allocating GPU memory, building CUDA graphs, kernel warmup, health checks, and route registration all take time—during which new replicas cannot serve real traffic.
3. Scale-down only looks at instantaneous low load, harming warm replicas. Reclaiming a freshly warmed replica during a traffic lull means a cold start when the next wave arrives, creating an oscillation cycle of “scale up, scale down, scale up.”

Therefore, the goal of LLM Serving autoscaling is not “keep the GPU fully utilized,” but rather: meet SLOs stably within given cost constraints, and avoid introducing new jitter from the scaling actions themselves.

---

Core Principles: Treat Autoscaling as a Control Loop

A production-ready LLM Serving autoscaling system needs at least four types of signals.

1. Queue Depth

Queue depth is the earliest signal of trouble. User experience degradation usually starts with queuing, not when GPU utilization hits 100%.

Key metrics include:

• Ingress queue length and wait time distribution
• pending_requests and ongoing_requests per replica
• Internal scheduling queue depth within a replica

Ray Serve’s autoscaling documentation explicitly recommends using target_ongoing_requests and max_ongoing_requests to define a steady state, making scaling decisions based on ongoing requests per replica. In the LLM context, queue depth reflects user experience more directly than CPU.

2. P95 / P99 Latency

LLM service latency observation must be broken down into three dimensions:

Metric	Meaning	Description
TTFT	Time To First Token	Wait time before the user sees the first token, directly impacting interactive feel
TPOT	Time Per Output Token	Interval between each token during generation, reflecting decode efficiency
E2E Latency	End-to-End Latency	Covers gateway, queuing, prefill, decode, streaming output, and post-processing

Relying only on average latency masks long-tail issues. Scaling strategies should at least use P95 as a protection signal, and optionally set stricter P95 targets for high-priority tenants.

3. Token Throughput

The real load for an LLM service isn’t QPS, it’s token traffic. Key metrics include:

• input_tokens/s and output_tokens/s
• prefill_tokens/s and decode_tokens/s
• max_tokens limit per request
• Token budget consumption rate per tenant

When the number of requests stays the same but the average prompt length increases, QPS won’t trigger any alert, but prefill pressure has already risen significantly. Token throughput is a more truthful load metric than QPS.

4. GPU Utilization and Memory Pressure

GPU utilization is still important, but should be used as a capacity and safety boundary signal, not the sole control target. You need to monitor simultaneously:

• GPU utilization (SM occupancy)
• GPU memory used
• KV cache or memory fragmentation pressure
• OOM / retry / preemption counts
• Remaining capacity per replica to accept new requests

NVIDIA Triton’s rate limiter emphasizes: in multi-model or resource-constrained scenarios, “being able to accept a request” and “should execute a request immediately” are not the same problem. The rate limiter controls the request scheduling rate to prevent running too many instances concurrently and overloading the server.

---

Recommended Architecture: Three-Layer Separation of Ingress Throttling, Queue Observation, and Autoscaler Controller

Don’t embed autoscaling logic inside the model serving process itself. A more robust approach is to split it into three layers.

Layer 1: Gateway / Ingress

The ingress layer handles tenant identification, authentication, rate limiting, request size checks, and routing. The key is to attach a cost profile to each request:

{
  "tenant_id": "tenant-a",
  "model": "llama-70b-chat",
  "input_tokens_estimated": 4200,
  "max_output_tokens": 800,
  "priority": "interactive",
  "request_class": "chat_long_context"
}

This profile isn’t meant to replace the model service; it provides a unified semantic for queuing, rate limiting, cost attribution, and autoscaling.

Layer 2: Serving Pool

The serving pool can be vLLM, Triton, Ray Serve, a custom inference service, or a managed endpoint. Each replica must expose sufficiently rich metrics:

metrics:
  request_queue_depth: gauge
  ongoing_requests: gauge
  ttft_p95_ms: histogram
  tpot_p95_ms: histogram
  input_tokens_per_second: counter
  output_tokens_per_second: counter
  gpu_utilization: gauge
  gpu_memory_used_ratio: gauge
  rejected_requests_total: counter
  oom_total: counter

If metrics are limited to GPU utilization and HTTP latency, the autoscaler controller will struggle to decide whether to scale up, throttle, split long requests, or fix a slow replica.

Layer 3: Autoscaler Controller

The controller should not act on a single metric directly, but compute a composite state:

autoscalingPolicy:
  minReplicas: 2
  maxReplicas: 24
  warmPoolReplicas: 2
  scaleUp:
    queueDepthPerReplica: 8
    p95LatencyMs: 2500
    gpuUtilization: 0.78
    consecutiveWindows: 2
  scaleDown:
    queueDepthPerReplica: 1
    p95LatencyMs: 1200
    gpuUtilization: 0.35
    consecutiveWindows: 10
  cooldown:
    scaleUpSeconds: 30
    scaleDownSeconds: 300
  safety:
    maxNewReplicasPerStep: 4
    maxHourlyGpuCostUsd: 120
    rejectWhenQueueWaitMsAbove: 8000

This configuration expresses three core principles:

1. Scale up faster than scale down, but not infinitely fast. Add at most 4 replicas per step, with a 30-second scale-up cooldown.
2. Scale down must be more conservative. LLM replicas have high cold-start costs. Use a 300-second scale-down cooldown and require 10 consecutive windows of satisfied conditions before acting.
3. When scale-up fails, enter protection mode. Reject low-priority requests, reduce max output tokens, or switch to a smaller model.

---

Engineering Implementation: A Production-Ready Scaling Decision Flow

Step 1: Define a Capacity Unit

Don’t say “a replica can handle X QPS.” Define a capacity unit closer to LLM reality:

capacity_unit = weighted_input_tokens + weighted_output_tokens + queue_penalty

A practical implementation:

def estimate_request_cost(
    input_tokens: int,
    max_output_tokens: int,
    request_class: str,
) -> float:
    prefill_weight = 1.0
    decode_weight = 2.0
    class_factor = {
        "chat_short": 1.0,
        "chat_long_context": 1.4,
        "agent_tool_loop": 1.6,
        "batch_summary": 0.8,
    }.get(request_class, 1.0)
    return class_factor * (
        input_tokens * prefill_weight + max_output_tokens * decode_weight
    )

This isn’t precise modeling; it allows the gateway, queue, and controller to communicate using the same load semantics.

Step 2: Use “OR” for Scale-Up Signals, “AND” for Scale-Down Signals

Scale-up is about protecting user experience. So if any one of multiple risk signals triggers persistently, you can scale up:

• Queue depth exceeds threshold for two consecutive windows
• P95 latency exceeds SLO for two consecutive windows
• GPU utilization and memory pressure are both near their limits
• input_tokens/s or output_tokens/s suddenly spikes

Scale-down must be more conservative, requiring multiple signals to be satisfied simultaneously: low queue, stable latency, low GPU, no pending long requests, and no traffic observation period for a newly deployed version.

Kubernetes HPA’s basic algorithm calculates the target number of replicas based on the ratio of current metrics to target metrics. LLM Serving can reuse this idea, but metric selection must expand from CPU to queue depth, latency, and token throughput.

Step 3: Introduce a Warm Pool

The cold-start cost of large models is much higher than for standard web services. Even if the container starts quickly, the following steps can prevent a new replica from serving real traffic for a while:

• Model weight loading (tens of GB)
• GPU memory allocation and defragmentation
• CUDA graph construction and kernel warmup
• Route registration and health checks

Therefore, production systems should maintain a warm pool—replicas that have completed model loading and health checks but are not yet receiving primary traffic. When the ingress queue starts to rise, first cut over to the warm pool, then trigger new replicas to replenish the warm pool level.

Step 4: Put Admission Control Before Scaling

Autoscaling is not infinite capacity. During traffic spikes, scaling may not be fast enough. Protecting the system from crashing is more important than trying to scale:

Protection Action	Use Case
Return 429 or queuing prompt for low-priority requests	Instantaneous traffic spikes
Limit max input/output tokens per request	Abnormally long requests
Pause non-real-time batch requests	Resource contention
Route to a smaller or degraded model	Primary model overloaded
Enable semantic caching for identical prompts	Repeated requests
Split ultra-long context requests into a dedicated pool	Long-tail requests impacting short ones

Core idea: Rejecting some requests is more controllable than letting all requests time out together.

Step 5: Make Scaling Actions Explainable

Every scale-up or scale-down should leave a decision record:

{
  "time": "2026-07-08T23:02:17Z",
  "model_pool": "chat-70b-prod",
  "action": "scale_up",
  "from_replicas": 8,
  "to_replicas": 12,
  "reason": [
    "queue_depth_per_replica=11 > 8",
    "p95_latency_ms=3100 > 2500",
    "gpu_utilization=0.81 > 0.78"
  ],
  "cooldown_seconds": 30,
  "cost_guardrail": "within_budget"
}

Without decision records, subsequent troubleshooting is guesswork. Especially when business stakeholders question cost increases, you must be able to answer: which SLO risk triggered the scale-up, how long did it last, were there alternative actions, and did it cross the cost boundary?

---

Applicable Scenarios

Suitable scenarios:

• Online Chat, Copilot, Agent, or customer service assistants—sensitive to P95 and TTFT
• Multi-tenant model serving—needs to prevent a single large tenant from saturating the shared GPU pool
• Hosting multiple models or versions on the same cluster—avoids contention for memory and execution slots
• Clear traffic peaks and valleys—balances stable experience with GPU cost

Unsuitable scenarios:

• Purely offline batch processing—prioritize task queues, batch windows, and cost optimization; sub-second autoscaling is often unnecessary
• Highly real-time voice or video conversations—scaling must rely on pre-warmed capacity and traffic prediction, not just reacting to queue increases

---

Common Misconceptions

Misconception 1: Higher GPU Utilization is Always Better

High GPU utilization usually means resources are fully used, but it doesn’t equal good user experience. For interactive LLM services, over-pursuing GPU utilization can sacrifice TTFT and P95. Layer by business need: interactive requests prioritize latency, offline requests prioritize throughput.

Misconception 2: Scaling Can Fix All Slow Requests

Scaling solves capacity shortages, not single requests that are too long, inherently slow models, network issues, slow downstream tools, overly large prompts, or streaming output blocking. Profile slow requests before scaling to avoid masking architectural problems with more GPUs.

Misconception 3: Scale Down Based Only on Low Load Windows

LLM service scale-down must consider cold-start costs and the risk of the next traffic wave. Use longer observation windows, sufficient downscale cooldown, and a minimum warm pool for protection.

Misconception 4: All Requests Share One Pool

Mixing short prompts, long contexts, Agents, multimodal, and batch tasks in one pool is the easiest way to cause mutual interference. In production, split into multiple pools by model, request type, tenant tier, and latency tier, each with its own scaling policy.

---

Production Readiness Checklist

Metric Completeness

• Do you have ingress queue, per-replica queue, and ongoing requests?
• Are TTFT, TPOT, and end-to-end latency distinguished?
• Are input_tokens/s and output_tokens/s recorded?
• Do you have GPU utilization, memory pressure, and OOM counts?
• Are metrics tagged by tenant, model, and request type?

Scaling Strategy

• Are minReplicas, maxReplicas, and warm pool configured?
• Are different observation windows configured for scale-up and scale-down?
• Are cooldown and hysteresis in place to avoid oscillation?
• Is the maximum number of new replicas per step limited?
• Is there a cost cap and anomaly alert?

Protection Mechanisms

• Can the ingress rate-limit by tenant?
• Can low-priority requests be rejected or degraded?
• Can the maximum tokens per request be limited?
• Can batch and interactive requests be separated into different pools?
• Is there a fallback when scale-up fails?

Replay Validation

• Have you replayed historical peak traffic?
• Have you simulated new replica cold-start delays?
• Have you tested scenarios like insufficient GPU, unschedulable nodes, and image pull failures?
• Have you validated the stability of consecutive scale-ups and scale-downs?
• Can you explain every scaling action?

---

References

• Kubernetes Documentation: Horizontal Pod Autoscaling
• Ray Serve Documentation: Advanced Autoscaling
• NVIDIA Triton Inference Server: Rate Limiter
• Kubernetes Documentation: Scaling on Custom Metrics