LLM Inference Capacity Planning in Production: Estimating GPU Headroom with Token Throughput Stress Tests and Queue Depth
A practical guide to transforming LLM inference capacity from reactive scaling to measurable, estimable, and alertable capacity engineering. Covers token throughput, queue depth, P99 TTFT, GPU headroom, and a pre-launch checklist to help teams identify bottlenecks and make quantitative scaling decisions.