Tag: inference-optimization

19 articles

• LLM Serving CUDA Graph in Production: Taming Kernel Launch Jitter with Shape Buckets, Piecewise Capture, and Warmup Plans

  CUDA Graphs cut LLM inference launch overhead, but dynamic batching, shape variance, and first-capture costs create latency jitter. This guide covers shape buckets, piecewise capture, warmup strategies, memory trade-offs, and production rollback gates using vLLM and TensorRT-LLM to fix host-side bottlenecks when GPUs aren't saturated but latency won't drop.

  09/03/2026 llm-serving / cuda-graph / inference-optimization / vllm / tensorrt-llm

• Multimodal LLM Serving in Production: Taming Vision Request Tail Latency with Parallel Media Decode, Encoder Cache, and Encoder Disaggregation

  After deploying multimodal LLMs, vision request tail latency often spikes due to image download, decode, vision encoding, and repeated processing. This article combines Dynamo, vLLM, and EPD practices to provide production governance methods including media decode offloading, Encoder Cache, dedicated Encoder Workers, input budgets, and stage-level observability.

  08/29/2026 Multimodal LLM Serving / Vision Language Model / Encoder Cache / Inference Optimization / Tail Latency Governance

• Serverless GPU Inference Cold Start Mitigation: Reducing First-Token Latency with Warm Pools and Staged Pre-Warming

  An in-depth analysis of the six-stage bottleneck of LLM cold starts in serverless GPU scenarios, covering warm pool tiering strategies, staged pre-warming mechanisms, engineering architecture, and metrics to help teams find a verifiable balance between GPU cost and first-token latency.

  07/01/2026 LLM Serving / Serverless GPU / Cold Start / Warm Pool / Inference Optimization / GPU Inference / Cold Start Optimization

• Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

  A systematic guide on how Continuous Batching improves LLM inference throughput via iteration-level scheduling, covering static batching pain points, scheduling mechanics, KV Cache constraints, key parameters, monitoring metrics, and deployment pitfalls.

  07/01/2026 LLM Serving / Continuous Batching / Inference Optimization / GPU / vLLM

• Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

  A systematic deep dive into Continuous Batching for LLM inference serving—covering principles, scheduling, engineering deployment, monitoring metrics, and go-live checklists to reduce queuing waste and maximize GPU throughput.

  07/01/2026 LLM Serving / Continuous Batching / Inference Optimization / KV Cache / MLOps

• PagedAttention in Production: Boosting LLM Serving Throughput with Paged KV Cache

  A systematic guide to PagedAttention principles, paged KV Cache management, continuous batching synergy, monitoring metrics, and pre-deployment checks to reduce GPU memory fragmentation and boost throughput in LLM inference serving.

  07/01/2026 LLM / LLM Serving / PagedAttention / KV Cache / Inference Optimization

• Prefix Caching in Production: Reducing TTFT with KV Cache Reuse

  A comprehensive guide to Prefix Caching for LLM inference: principles, suitable workloads, engineering integration, monitoring, security boundaries, and common pitfalls for RAG, Agent, and multi-turn dialogue applications.

  07/01/2026 LLM / Inference Optimization / KV Cache / Prefix Caching / Engineering Practices

• Speculative Decoding in Production: Reducing LLM Decode Latency with Draft-Verify

  A systematic guide to Speculative Decoding's Draft-Verify mechanism, covering vLLM and TensorRT-LLM integration, metric tuning, applicability boundaries, and production deployment checks to reduce decode latency and avoid overhead from low acceptance rates.

  07/01/2026 LLM / Inference Optimization / Speculative Decoding / LLM Serving / Engineering Practice

• KV Cache Quantization in Production: Reducing Long-Context Inference Memory with FP8/INT8

  A systematic guide to KV cache quantization for long-context LLM inference: value, principles, vLLM/TensorRT-LLM implementation, evaluation metrics, and production risks to help teams safely reduce memory and latency costs.

  07/01/2026 LLM / Inference Optimization / KV Cache / Quantization / LLMOps / vLLM / TensorRT-LLM

• Continuous Batching in Production: Keeping LLM Services GPU-Saturated Under High Concurrency

  A systematic guide to continuous batching for LLM serving: core principles, scheduling strategies, KV Cache constraints, engineering deployment steps, and monitoring metrics. Covers the full path from proof-of-concept to production, helping teams boost throughput while controlling tail latency in high-concurrency scenarios.

  07/01/2026 LLM Serving / Continuous Batching / vLLM / KV Cache / Performance Engineering / Inference Optimization / GPU Scheduling

• Speculative Decoding in Production: Reducing LLM Latency with Draft Models

  A systematic guide to Speculative Decoding for LLM inference: principles, engineering trade-offs, model selection, monitoring metrics, and common pitfalls. Learn when and how to deploy draft models for latency reduction.

  06/30/2026 LLM / Inference Optimization / Speculative Decoding / LLMOps / Performance Engineering

• Prefix Caching in Production: A Complete Guide to Reusing KV Cache for RAG and Agent Services

  A systematic guide to Prefix Caching for reusing KV Cache of shared prompts and contexts, reducing TTFT, inference costs, and redundant prefill in RAG, Agent, and long-document QA scenarios. Includes engineering checklists for vLLM, OpenAI API, and SGLang.

  06/30/2026 LLM Serving / KV Cache / RAG / Agent Infrastructure / Prefix Caching / Inference Optimization

• Prefill-Decode Disaggregation in Practice: Governing TTFT and TPOT for Long-Context LLM Serving

  A practical guide to Prefill-Decode disaggregation for LLM serving: principles, production architecture, monitoring, and a go-live checklist to govern TTFT and TPOT under long-context, high-concurrency workloads.

  06/30/2026 LLM Serving / Prefill Decode / Inference Optimization / KV Cache / SLO

• Continuous Batching in Practice: Why Traditional Fixed Batching Falls Short for LLM Serving

  From fixed batching and iteration-level scheduling to PagedAttention, this article systematically explains the principles, benefits, trade-offs, and production deployment checklist for Continuous Batching in LLM inference serving.

  06/30/2026 LLM Serving / Continuous Batching / PagedAttention / Inference Optimization

• MoE Expert Parallelism in Practice: Communication and Load Balancing for Sparse LLM Inference

  A practical guide to Expert Parallelism for MoE inference: communication overhead, load balancing, parallel strategy combinations, and a pre-deployment checklist for production.

  06/30/2026 LLM Serving / Mixture of Experts / Expert Parallelism / Inference Optimization

• Multi-LoRA Serving in Practice: Serving Multiple Business Adapters with One Base Model

  A practical guide to Multi-LoRA Serving: how to share one base model across multiple LoRA adapters for cost-effective, multi-tenant LLM inference.

  06/30/2026 LLM Serving / LoRA / PEFT / Inference Optimization

• Chunked Prefill in Production: Latency Governance for Long-Context Inference

  A deep dive into how Chunked Prefill and Continuous Batching work together to manage latency in long-context LLM inference, covering scheduling principles, chunk size tuning, production metrics, and a deployment checklist.

  06/29/2026 LLM Serving / Chunked Prefill / Continuous Batching / Inference Optimization

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift LLM inference bottlenecks from single-card throughput to tail-latency governance. This post dives into Prefill-Decode disaggregation, KV cache transfer, resource planning, and a pre-launch checklist to move LLM serving from 'working' to stable and scalable.

  06/29/2026 LLM Serving / Prefill Decode / KV Cache / Inference Optimization / Long Context / Tail Latency / RAG / Agent

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift the LLM inference bottleneck from single-GPU throughput to tail-latency management. This article dives into the principles, trade-offs, KV cache transfer, resource planning, and a go-live checklist for Prefill-Decode disaggregation.

  06/29/2026 LLM Serving / Prefill-Decode / KV Cache / Inference Optimization / Long Context / Latency Management