LLM Serving Autoscaling in Production: Safe Scaling with Queue Depth, P95 Latency, and GPU Utilization
A deep dive into production-grade autoscaling for LLM serving. Build a multi-layer control loop using queue depth, P95 latency, token throughput, and GPU utilization, with warm pools, admission control, cost boundaries, and cooldown strategies to meet inference SLOs under cost constraints.