LLM Inference Capacity Planning in Production: Estimating GPU Headroom with Token Throughput Stress Tests and Queue Depth

The Core Problem: LLM Capacity Isn’t “QPS × Average Latency”

When doing capacity planning for traditional web services, teams typically start from QPS, average latency, and CPU utilization, using simple linear models to estimate required resources. But large model online inference is fundamentally different:

• Request cost is highly uneven. A customer service Q&A with 200 input tokens and 50 output tokens, and an analysis task with 8K input tokens and 2K output tokens, do not occupy the GPU on the same scale.
• Inference is divided into prefill and decode phases. The former primarily processes the input context and is compute-intensive; the latter generates output tokens one by one and is memory-bandwidth intensive. The resource bottlenecks for each phase are different.
• The root cause of tail latency isn’t just slow model computation. It more often comes from queue wait times, batching congestion, KV cache pressure, and resource consumption by long-output requests.

Therefore, LLM capacity planning cannot just ask, “How many QPS can this model handle?” A more accurate question is: Given a specific distribution of input/output tokens, P99 TTFT targets, P99 end-to-end latency targets, and peak traffic patterns, how much additional load can the current GPU pool sustain? When must we scale up? And when is the problem simply that scheduling parameters, queue strategies, or rate-limiting policies aren’t configured correctly?

Core Principles: Decomposing Capacity into Tokens, Queues, and SLOs

1. Replace Average Requests with Token Distributions

The input for capacity planning should not be a single “average request.” You should at least record the following distributions:

Field	Purpose
input_tokens	Determines prefill pressure and initial KV cache usage
output_tokens	Determines decode duration and user wait time
total_tokens	Used to estimate context window and KV cache pressure
arrival_rate	Requires not just the mean, but also peaks, spikes, and periodicity
tenant / scenario	Model traffic from different businesses separately to prevent short Q&A from masking long tasks

NVIDIA’s GenAI-Perf includes LLM metrics like TTFT, inter-token latency, request latency, input/output sequence length, output token throughput, and request throughput, and supports generating load based on concurrency or request rate. These metrics are far more suitable as an LLM inference capacity baseline than simple QPS.

2. Explain Tail Latency with Queue Depth

When a GPU pool is nearing saturation, the user-perceived latency doesn’t always show up first in TPOT. It shows up first in queue time and TTFT — new requests must wait for scheduling slots, KV cache, batching windows, or replica resources.

Production metrics exposed by vLLM allow you to decompose “the model is slow” into specific causes:

num_requests_running          # Number of currently running requests
num_requests_waiting           # Number of currently queued requests
kv_cache_usage_perc            # KV Cache usage percentage
request_queue_time_seconds     # Request queue time
request_prefill_time_seconds   # Prefill phase duration
request_decode_time_seconds    # Decode phase duration
time_to_first_token_seconds    # Time to First Token (TTFT)

With these metrics, teams can upgrade from the vague judgment of “the model is slow” to a precise diagnosis of “slow queuing, slow prefill, slow decode, KV cache pressure, or scheduling congestion.”

3. Use P99 TTFT as the First Capacity Red Line

For scenarios like chat, search-augmented Q&A, and agent control planes, TTFT often reflects user experience better than the full response time. A TTFT violation usually means congestion has already occurred at the entry point, in the queue, or during the prefill phase.

Capacity planning can adopt the following principle:

Available Capacity = Maximum stable load when P99 TTFT and P99 queue time are within SLO Safety Headroom = Available Capacity − Expected Peak Load Scale-Up Threshold = Safety Headroom falls below a business-defined ratio, or SLO burn rate exceeds limits over a continuous window

The key is not to chase a fixed GPU utilization percentage, but to find the inflection point where latency starts to increase non-linearly. Many inference services see unacceptable P99 latency long before GPU utilization is near 100%. Conversely, in some scenarios, GPU utilization may not appear high, but the system is overwhelmed by long-tail requests, KV cache pressure, memory fragmentation, or single-tenant spikes.

Engineering Implementation: From a Single Stress Test to a Reusable Capacity Model

Step 1: Collect Real Traffic Footprints

Don’t directly use random prompts to stress-test production capacity. A better approach is to anonymously collect capacity-related fields from real requests:

{
  "timestamp_ms": 1730000000000,
  "tenant": "risk-review",
  "scenario": "long_report_qa",
  "model": "llama-70b",
  "input_tokens": 6420,
  "output_tokens": 820,
  "stream": true,
  "priority": "normal"
}

Note that only capacity fields are needed here; the original sensitive prompt should not be stored. Token length, arrival time, model, tenant, and scenario are already sufficient for most capacity assessments.

Step 2: Build a Stress Test Matrix

The stress test matrix should include at least four types of scenarios, not just a single curve:

Scenario	Goal	Metrics to Watch
Baseline Stress Test	Find the stable throughput of a single replica or GPU pool	TTFT, TPOT, request latency, tokens/s
Peak Stress Test	Verify headroom during business peaks	P95/P99 queue time, P99 TTFT
Long-Tail Stress Test	Verify the impact of long inputs, long outputs, and agent tasks	KV cache pressure, decode time, wait queue
Mixed Traffic Replay	Verify capacity inflection points under real-world conditions	SLO attainment rate, tenant fairness, error rate

NVIDIA GenAI-Perf supports synthetic input, reading datasets from files, and specifying input/output lengths, request counts, concurrency, and request rates. For production teams, the most valuable approach is “real traffic replay + peak amplification”, not just running fixed-length short prompts.

Step 3: Convert Stress Test Results into Capacity Curves

Don’t let the output of a single stress test remain in a report. It’s recommended to organize the results into a capacity curve:

Model: llama-70b
Hardware: 8 × H100
Scenario: mixed_chat_agent
SLO: P99 TTFT ≤ 800ms, P99 E2E ≤ 12s

Stable Zone: 0–120 req/s      # All SLOs met, queue depth stable
Risk Zone: 120–150 req/s       # Queue time starts to rise, TTFT variance increases
Unstable Zone: >150 req/s      # P99 TTFT increases non-linearly, SLOs continuously violated

Recommended Production Watermark: ≤100 req/s    # Reserve ~20% peak headroom

This curve is far more useful than “average QPS 150” because it tells operations and business teams: when it’s safe, when it’s only briefly tolerable, and when it will lead to uncontrolled tail latency.

Step 4: Integrate Queue Metrics into Auto-Scaling

If deployed on Ray Serve, Kubernetes, or similar platforms, scaling metrics should not rely solely on CPU or GPU utilization. Ray Serve’s autoscaling supports controlling the desired concurrency per replica via target_ongoing_requests; Kubernetes HPA also supports horizontal scaling based on custom metrics.

LLM services are better suited to introducing the following custom scaling signals:

autoscaling_signals:
  - p99_time_to_first_token_seconds
  - p99_request_queue_time_seconds
  - num_requests_waiting
  - kv_cache_usage_perc
  - prompt_tokens_per_second
  - generation_tokens_per_second
  - slo_burn_rate

Scale-down policies should be more conservative than scale-up policies. LLM model replicas typically involve weight loading, warm-up, KV cache establishment, and scheduling stabilization. Scaling down too quickly can cause the next wave of traffic to trigger a cold start again. Therefore, cooldown periods, stabilization windows, and warm pools are necessary.

Capacity Estimation Formula: Start Rough, Refine with Stress Tests

A rough model can be used for the first round of estimation:

Peak Input Tokens/s  = Peak Requests/s × P95 input_tokens
Peak Output Tokens/s = Peak Requests/s × P95 output_tokens

Required GPU Count ≈ max(
  Peak Input Tokens/s / Stable Prefill Tokens/s per GPU,
  Peak Output Tokens/s / Stable Decode Tokens/s per GPU
) × Safety Factor

⚠️ This formula is only for estimation and should not be used directly as a basis for procurement or launch. It ignores the effects of queuing, batching, KV cache, mixed request lengths, tenant spikes, and hardware failures. The inference-fleet-sim paper also points out that LLM inference GPU fleet sizing depends on the full token-length distribution, routing policy, and queueing dynamics, and many scenarios lack a simple closed-form answer.

Therefore, the recommended process is:

1. Use the rough model to get a first version of the GPU count.
2. Run stress tests with the real token distribution to find the SLO inflection point.
3. Use queue metrics to diagnose whether the bottleneck is prefill, decode, KV cache, or scheduling.
4. Apply a safety factor based on peak traffic and business growth projections.
5. After launch, continuously refine the capacity curve with real metrics.

Applicable Scenarios

This approach is particularly suitable for:

• Self-built online inference platforms using vLLM, Triton, TensorRT-LLM, SGLang, or Ray Serve.
• Multi-business environments sharing a single GPU pool, where you need to answer, “How much new business can we still take on?”
• Cases where P99 TTFT is already rising, but GPU utilization doesn’t explain the problem.
• Teams preparing to move from a single-model pilot to multi-tenant production, needing to define scaling, rate-limiting, and budgeting strategies.
• Situations requiring a quantitative assessment before purchasing GPUs, selecting cloud instances, or planning baseline capacity.

Common Pitfalls

Pitfall 1: Only Looking at Average QPS

Average QPS masks the long tail. The cost of an LLM request is determined by token length, output length, context caching, and arrival time. Capacity planning should at least look at the P50, P95, and P99 distributions of input and output tokens.

Pitfall 2: Higher GPU Utilization is Always Better

Inference services are not offline training. Online services need headroom for spikes, long tails, and failure recovery. When GPU utilization is too high, queue wait times amplify rapidly, and P99 TTFT can spiral out of control before the average latency does.

Pitfall 3: Autoscaling Can Replace Capacity Planning

Autoscaling is a runtime adjustment, not capacity planning itself. If the maximum replica count, node pool, model loading time, and GPU quota are insufficient, autoscaling can only expose the bottleneck faster; it cannot create capacity out of thin air.

Pitfall 4: Short Prompt Stress Tests Represent All Business

Short Q&A stress tests typically overestimate platform capacity. Agents, code analysis, long-document Q&A, multi-turn conversations, and long-output reports can all significantly change the prefill/decode ratio and KV cache pressure.

Pre-Launch Checklist

Metric Completeness

Before launch, confirm that the following metrics are being collected:

• P50/P95/P99 TTFT
• P50/P95/P99 request latency
• P95/P99 request queue time
• prompt tokens/s and generation tokens/s
• input_tokens, output_tokens distribution
• running requests, waiting requests
• KV cache usage percentage
• GPU memory, SM, power, and error metrics
• SLO attainment rate broken down by tenant, model, and scenario

Stress Test Completeness

Stress tests cannot just run a single curve. They must cover at least low-traffic, normal, peak, spike, long-tail, and mixed traffic scenarios. Every time the model version, context length, batching parameters, quantization method, or hardware specification changes, the capacity curve should be regenerated.

Scale-Up Strategy

capacity_policy:
  safe_utilization_window: "P99 TTFT within SLO, no sustained increase in queue time"
  scale_up_trigger:
    - "p99_ttft > slo_threshold for 3 windows"
    - "num_requests_waiting is continuously increasing"
    - "kv_cache_usage_perc > 0.85"
  scale_down_rule:
    cooldown: "10m–30m"
    require_low_queue: true
    require_stable_slo: true
  reserve:
    peak_headroom: "20%–30%"
    failure_headroom: "Calculated separately based on node failure and repair time"

Release Gate

Before launching a new model or new parameters, you must confirm:

• The new version does not lower the capacity inflection point under the same traffic replay.
• P99 TTFT and queue time have not significantly worsened.
• Long-input and long-output scenarios do not squeeze the experience of short requests.
• Scaling down and then scaling up again does not introduce unacceptable cold-start latency.
• Rate limiting, degradation, and queue upper limits are configured and have observable alerts.

FAQ

How often should LLM capacity planning be done?

At a minimum, redo it whenever the model version, context length, inference engine, quantization method, hardware specification, or business traffic structure changes. For rapidly growing platforms, it’s recommended to refresh the capacity curve weekly or bi-weekly using real traffic.

Which is more important: P99 TTFT or P99 E2E latency?

Both are important. TTFT is better for detecting entry-point queuing and prefill congestion, while E2E latency is better for detecting long outputs and decode pressure. Chat experiences typically prioritize TTFT, while report generation and batch analysis focus more on E2E latency and completion rate.

How do I stress test without real production traffic?

You can start by building a rough curve using synthetic input tokens and output tokens, but you must prepare several business assumptions: short Q&A, long documents, long outputs, and multi-turn agent interactions. After launch, replace the synthetic data with anonymized real token distributions as soon as possible.

References

• NVIDIA Triton Inference Server — GenAI-Perf Documentation
  https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html

• vLLM — Production Metrics
  https://docs.vllm.ai/en/latest/usage/metrics/

• Ray Serve — Autoscaling Guide
  https://docs.ray.io/en/latest/serve/autoscaling-guide.html

• Kubernetes — Horizontal Pod Autoscaling
  https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/

• inference-fleet-sim: A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference
  https://arxiv.org/abs/2603.16054