Tag: Tail Latency

2 articles

• LLM Capacity Load Testing in Production: Finding Real Safe QPS with Timed Traces, Burstiness, and Tail-SLO Gates

  LLM capacity testing shouldn't rely on fixed concurrency and average latency alone. This article combines vLLM, AIPerf, and MLPerf to explain Timed Traces, burst traffic, tail-latency gates, and Safe QPS — a production-grade method for finding the true capacity inflection point before launch.

  09/02/2026 LLM Serving / Capacity Testing / Benchmark / Tail Latency / Safe QPS / Timed Trace / Burstiness / Capacity Planning

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift LLM inference bottlenecks from single-card throughput to tail-latency governance. This post dives into Prefill-Decode disaggregation, KV cache transfer, resource planning, and a pre-launch checklist to move LLM serving from 'working' to stable and scalable.

  06/29/2026 LLM Serving / Prefill Decode / KV Cache / Inference Optimization / Long Context / Tail Latency / RAG / Agent