Tag: cuda-graph

2 articles

• LLM Serving CUDA Graph in Production: Taming Kernel Launch Jitter with Shape Buckets, Piecewise Capture, and Warmup Plans

  CUDA Graphs cut LLM inference launch overhead, but dynamic batching, shape variance, and first-capture costs create latency jitter. This guide covers shape buckets, piecewise capture, warmup strategies, memory trade-offs, and production rollback gates using vLLM and TensorRT-LLM to fix host-side bottlenecks when GPUs aren't saturated but latency won't drop.

  09/03/2026 llm-serving / cuda-graph / inference-optimization / vllm / tensorrt-llm

• LLM CUDA Graph Production Practices: Reducing Decode Launch Overhead with Shape Buckets, Piecewise Capture, and Eager Fallback

  CPU kernel launch overhead in LLM decode is often overlooked. This deep dive explains CUDA Graph capture and replay, combined with shape bucket design, piecewise capture, warmup state machines, and eager fallback, delivering a production-ready solution to reduce tail latency and P99 jitter.

  07/11/2026 LLM Inference / CUDA Graph / GPU Inference / vLLM / Performance Optimization