Tag: SLO

3 articles

• LLM Inference Capacity Planning in Production: Estimating GPU Headroom with Token Throughput Stress Tests and Queue Depth

  A practical guide to transforming LLM inference capacity from reactive scaling to measurable, estimable, and alertable capacity engineering. Covers token throughput, queue depth, P99 TTFT, GPU headroom, and a pre-launch checklist to help teams identify bottlenecks and make quantitative scaling decisions.

  07/05/2026 LLM Serving / Capacity Planning / GPU Inference / SLO / vLLM / Token Throughput

• LLM Serving Overload Protection: SLO-Driven Autoscaling and Admission Control

  A practical guide to building overload protection for LLM inference services using SLO-driven autoscaling and admission control. Covers queue length, token pressure, TTFT/TPOT, and tenant quotas to prevent system collapse under traffic spikes.

  07/01/2026 LLM Serving / Autoscaling / SLO / Admission Control / Overload Protection / Inference Infrastructure

• Prefill-Decode Disaggregation in Practice: Governing TTFT and TPOT for Long-Context LLM Serving

  A practical guide to Prefill-Decode disaggregation for LLM serving: principles, production architecture, monitoring, and a go-live checklist to govern TTFT and TPOT under long-context, high-concurrency workloads.

  06/30/2026 LLM Serving / Prefill Decode / Inference Optimization / KV Cache / SLO