LLM Capacity Load Testing in Production: Finding Real Safe QPS with Timed Traces, Burstiness, and Tail-SLO Gates

Load test results for generative inference services are often misleading because of the testing methodology itself. The most common report looks like this: fix concurrency at 50 or 100, run for a few minutes, then record average TTFT, average throughput, and GPU utilization. Those numbers are fine for comparing single-node optimizations, but they don’t answer the most important production question — under real request arrival patterns, how much traffic can this cluster actually sustain?

The reason is that generative inference is not a fixed-latency RPC. Input length, output length, multi-turn behavior, inter-arrival times, and burstiness all change queueing, batching, and memory pressure. Two traffic patterns with the same average QPS — one arriving uniformly, the other bursting within seconds — put completely different stress on tail latency.

So the goal of capacity testing shouldn’t be finding “maximum throughput.” It should be finding Safe QPS: the highest request rate at which the system consistently meets its defined Tail SLOs under a representative workload.

Background: Why “Saturating the GPU” ≠ Knowing System Capacity

100% GPU utilization only tells you the hardware is busy — not that the system is busy “correctly.” The real measure of production capacity is how much external traffic the system can absorb while meeting business commitments. A service that pegs the GPU but has lost control of p99 tail latency does not have higher usable capacity than one that runs below saturation with stable tail latency.

This article combines vLLM, NVIDIA AIPerf, and MLPerf Server scenarios to clarify four things: why fixed concurrency overestimates capacity, why average length doesn’t represent a real workload, why Burstiness must be an independent variable, and how to use a Tail-SLO Gate to find the true capacity inflection point.

Core Principle 1: Distinguish Closed-Loop Concurrency from Open-Loop Arrival Rate

Why Fixed Concurrency Masks Overload

Fixed concurrency is essentially a closed-loop workload: the client only sends the next request after one completes. When the server slows down, the client slows down too, and the new-request injection rate drops accordingly.

This approach is useful for observing the experience “when N users are active simultaneously,” but it can overestimate true online capacity — because production traffic doesn’t automatically stop arriving just because your GPU got slower.

In contrast, an open-loop, request-rate-driven load keeps sending requests on a predetermined schedule. When the server can’t keep up, queueing, TTFT, and end-to-end tail latency are exposed honestly.

vLLM’s current bench serve supports --request-rate to control the request rate directly, and can generate arrival times with Poisson or Gamma distributions; a --burstiness value below 1 produces more bursty arrival patterns. AIPerf similarly supports constant, poisson, gamma, and other arrival patterns.

Production load testing should run at least two types of experiments:

1. Closed-loop concurrency sweep: observe per-request experience and GPU saturation range at different concurrency levels.
2. Open-loop request-rate sweep: observe when queueing spirals out of control and Tail SLOs break under a fixed external arrival rate.

Core Principle 2: Average Length ≠ Real Workload

LLM request cost is highly correlated with token length. Using only fixed samples like “1K input, 256 output” tends to flatten the long-context, short-answer, long-answer, and mixed traffic that exists in real business.

A better approach is to extract an anonymized workload profile from production or pre-production logs, retaining at least:

• request timestamps;
• input token length;
• output token length;
• session or workload class;
• hash information for reproducing prefix similarity when needed — never raw business text.

vLLM’s latest timed_trace dataset directly supports fields like timestamp, input_length, output_length, and hash_ids, and can use the time information in the trace for self-timed replay. AIPerf also supports fixed-schedule Trace Replay with timestamps.

A minimal anonymized trace might look like:

{"timestamp": 0,   "input_length": 1200, "output_length": 52, "hash_ids": [0, 1, 2]}
{"timestamp": 105, "input_length": 1800, "output_length": 26, "hash_ids": [0, 3, 4, 5]}
{"timestamp": 274, "input_length": 1300, "output_length": 52, "hash_ids": [1, 4, 6]}

AIPerf’s official example replays on the original timestamps with a fixed schedule:

aiperf profile \
  --model YOUR_MODEL \
  --endpoint-type chat \
  --streaming \
  --url localhost:8000 \
  --input-file custom_trace.jsonl \
  --custom-dataset-type mooncake_trace \
  --fixed-schedule

The value of such traces goes beyond “looking more like production.” They expose problems that fixed-length tests rarely surface: concentrated arrival of long contexts, session bursts, context-window overflows, and completely different queue shapes at the same average QPS.

Core Principle 3: Burstiness Must Be an Independent Variable

Capacity tests often sweep only QPS while ignoring Burstiness. In reality, an average of 20 QPS could mean one request every 50ms uniformly, or dozens of requests in quick succession followed by a long idle period.

vLLM’s benchmark supports tuning burstiness via the Gamma distribution; AIPerf also exposes arrival pattern and smoothness as parameters. This means the test matrix shouldn’t be one-dimensional on QPS — it should include at least:

• request rate;
• burstiness / arrival pattern;
• input/output length distribution;
• concurrency cap;
• workload class.

Present production capacity results as multiple curves, not a single number. For example, at the same 30 QPS, report p95/p99 for Poisson, a smoother Gamma, a burstier Gamma, and real Trace Replay. That’s how you learn whether the system is sensitive to burst traffic.

Core Principle 4: Use a Tail-SLO Gate to Find the Capacity Inflection Point — Not Maximum TPS

The capacity curve of a generative service typically has a clear inflection point: before a certain arrival rate, throughput grows with traffic; past the inflection point, throughput growth slows while queueing time and tail latency rise sharply.

Safe QPS should therefore be defined as: the highest sustainable request rate that simultaneously satisfies all business SLOs and stability gates, under a specified workload profile, test duration, and hardware configuration.

Gates should include at least:

Metric	Meaning
Success Rate	You can’t “buy throughput” with timeouts, rejections, or errors
TTFT p95/p99	The most direct perceived wait for interactive requests
ITL / TPOT p95/p99	Whether generation feels choppy
E2E Latency p95/p99	The complete request experience
Queue Time / Queue Depth	Whether backlog is building persistently
Output Tokens/s	The server’s effective production capacity
GPU / HBM / CPU	For explaining bottlenecks — not as final SLOs

Gate thresholds should come from your business scenario, not copied from another company’s absolute numbers. For example, a release rule might be expressed as:

capacity_gate:
  success_rate: ">= ${SUCCESS_SLO}"
  ttft_p95: "<= ${TTFT_P95_SLO}"
  ttft_p99: "<= ${TTFT_P99_SLO}"
  itl_p99: "<= ${ITL_P99_SLO}"
  e2e_p99: "<= ${E2E_P99_SLO}"
  queue_growth: "no_sustained_growth"

MLPerf’s Server scenario also uses random arrivals with a latency constraint, rather than reporting only offline throughput. Its DeepSeek-R1 documentation specifically notes that Server target QPS usually requires manual search; it’s sometimes around 80% of Offline QPS, but on some systems it can be below 50%. This is a clear demonstration that Offline throughput cannot be treated as online capacity.

Engineering in Practice: Building a Four-Layer Load Testing System

Layer 1: Micro Benchmark

Goal: compare individual configurations, such as TP/EP layouts, quantization methods, kernel versions, or GPU models. Use fixed input/output lengths and stable concurrency to minimize variables. This layer answers “which config is faster,” not “how much production traffic can this handle.”

Layer 2: Synthetic Capacity Sweep

Use multiple input/output buckets and sweep request rate and burstiness. Each point must include warmup, and per-request results must be saved — not just aggregate means. vLLM’s bench serve supports saving detailed request metrics and can output percentiles for TTFT, TPOT, ITL, and E2E. In CI or a performance experiment platform, save model version, inference engine version, GPU, parallel configuration, and parameters as metadata for every benchmark run.

Layer 3: Timed Trace Replay

Sample and anonymize real business traffic, preserving timestamps and token length distributions. Run A/A and A/B replays on the same trace to compare: inference engine upgrades, GPU model changes, parallelism strategy changes, model version changes, and scheduler parameter changes. The most important thing here is keeping the workload constant — otherwise it’s hard to tell whether a performance change came from the system or the traffic.

Layer 4: Soak + Burst Test

Short load tests miss memory fragmentation, connection pool issues, cache heat changes, and slow queue growth. Before final release, add sustained steady-state testing, and inject burst traffic segments during the steady state to verify the system can recover to its original queue depth and tail latency levels.

Turning Test Results into Capacity Planning

Don’t output isolated conclusions like “35 QPS per GPU.” Instead, output a Capacity Envelope:

• Interactive workload: Safe QPS = X;
• Long-context workload: Safe QPS = Y;
• Bursty workload: Safe QPS = Z;
• Corresponding GPU count, model configuration, and Tail SLOs;
• Reserve headroom for at least one tier of business growth and failure redundancy.

X/Y/Z should come from actual testing — never pre-hardcoded. Capacity planning must also define SLOs for N-1 or single-replica failure scenarios, because “just barely saturated in the healthy state” is not an operable production capacity.

Common Pitfalls

Pitfall	Correct Approach
Looking only at average TTFT	Focus on p95/p99, and analyze in buckets by input length, tenant, or business type
Treating max TPS under fixed concurrency as capacity	Fixed concurrency creates closed-loop feedback; it’s fine for performance scanning but insufficient for defining online Safe QPS
Using the same token length for all requests	At minimum use short/medium/long buckets; ideally use a real trace
Using ignore_eos throughput directly for business capacity	That result is only for configuration comparison, not a capacity conclusion
Testing only average traffic, not bursts	Real systems die on a 10-second peak, not a 1-hour average — Burstiness must be tested explicitly

Pre-Launch Checklist

• Completed both closed-loop concurrency and open-loop request-rate tests.
• Covered production input/output token length distributions, not just a single fixed length.
• Tested at least Poisson/burst traffic or real-timestamp Trace Replay.
• Saved p95/p99 TTFT, ITL/TPOT, E2E, success rate, and queue metrics.
• Identified the capacity inflection point where Tail SLOs begin to degrade noticeably — not just recorded max TPS.
• Safe QPS is determined by business SLO gates, with headroom for failures and growth.
• Benchmark artifacts record model version, inference engine version, GPU, parallelism parameters, and test data version.
• Completed sustained steady-state testing before release, and confirmed queues recover after bursts end.

References

1. vLLM — vllm bench serve: https://docs.vllm.ai/en/latest/cli/bench/serve/
2. NVIDIA AIPerf — Trace Replay with Mooncake Traces: https://docs.nvidia.com/aiperf/benchmark-modes/trace-replay-with-mooncake-traces
3. NVIDIA AIPerf — Command Line Options: https://docs.nvidia.com/aiperf/reference/command-line-options
4. NVIDIA AIPerf — Comprehensive LLM Benchmarking: https://docs.nvidia.com/aiperf/getting-started/ai-perf-comprehensive-llm-benchmarking
5. MLCommons — Llama 2 70B: An MLPerf Inference Benchmark for Large Language Models: https://mlcommons.org/2024/03/mlperf-llama2-70b/
6. MLPerf Inference — DeepSeek-R1: https://docs.mlcommons.org/inference/benchmarks/language/deepseek-r1/