Tag: tensorrt-llm

5 articles

• LLM Serving CUDA Graph in Production: Taming Kernel Launch Jitter with Shape Buckets, Piecewise Capture, and Warmup Plans

  CUDA Graphs cut LLM inference launch overhead, but dynamic batching, shape variance, and first-capture costs create latency jitter. This guide covers shape buckets, piecewise capture, warmup strategies, memory trade-offs, and production rollback gates using vLLM and TensorRT-LLM to fix host-side bottlenecks when GPUs aren't saturated but latency won't drop.

  09/03/2026 llm-serving / cuda-graph / inference-optimization / vllm / tensorrt-llm

• LLM Model Format Conversion in Production: Avoiding Silent Degradation with Tensor Mapping, QKV Reordering, and Layer-Wise Validation

  A systematic guide to parameter mapping, QKV reordering, shard indexing, layer-wise numerical validation, and rollback gates when converting LLM weights between Hugging Face, Megatron, and TensorRT-LLM. Build a repeatable, auditable release pipeline to prevent silent degradation, output drift, and parallel slicing errors.

  07/22/2026 LLM Engineering / Model Conversion / Megatron / TensorRT-LLM / Safetensors

• KV Cache Quantization in Production: Reducing Long-Context Inference Memory with FP8/INT8

  A systematic guide to KV cache quantization for long-context LLM inference: value, principles, vLLM/TensorRT-LLM implementation, evaluation metrics, and production risks to help teams safely reduce memory and latency costs.

  07/01/2026 LLM / Inference Optimization / KV Cache / Quantization / LLMOps / vLLM / TensorRT-LLM

• Speculative Decoding in Production: Why Low QPS Speeds Up but High Load Slows Down

  A production-oriented analysis of speculative decoding's benefit boundaries, covering draft models, verification, acceptance rate, request load, and batching, with a pre-deployment checklist.

  06/29/2026 LLM Inference / Speculative Decoding / vLLM / TensorRT-LLM / Performance Optimization

• LLM Quantization in Practice: FP8, INT4, AWQ, and Production Inference Trade-offs

  A systematic guide to LLM inference quantization: FP8, INT4, AWQ, GPTQ, KV Cache quantization, benefits, risks, selection methods, and a go-live checklist for production engineers.

  06/29/2026 LLM Inference / Quantization / vLLM / TensorRT-LLM / Model Serving