Tag: Performance Optimization

2 articles

• LLM CUDA Graph Production Practices: Reducing Decode Launch Overhead with Shape Buckets, Piecewise Capture, and Eager Fallback

  CPU kernel launch overhead in LLM decode is often overlooked. This deep dive explains CUDA Graph capture and replay, combined with shape bucket design, piecewise capture, warmup state machines, and eager fallback, delivering a production-ready solution to reduce tail latency and P99 jitter.

  07/11/2026 LLM Inference / CUDA Graph / GPU Inference / vLLM / Performance Optimization

• Speculative Decoding in Production: Why Low QPS Speeds Up but High Load Slows Down

  A production-oriented analysis of speculative decoding's benefit boundaries, covering draft models, verification, acceptance rate, request load, and batching, with a pre-deployment checklist.

  06/29/2026 LLM Inference / Speculative Decoding / vLLM / TensorRT-LLM / Performance Optimization