Tag: vLLM

11 articles

• LLM Inference Capacity Planning in Production: Estimating GPU Headroom with Token Throughput Stress Tests and Queue Depth

  A practical guide to transforming LLM inference capacity from reactive scaling to measurable, estimable, and alertable capacity engineering. Covers token throughput, queue depth, P99 TTFT, GPU headroom, and a pre-launch checklist to help teams identify bottlenecks and make quantitative scaling decisions.

  07/05/2026 LLM Serving / Capacity Planning / GPU Inference / SLO / vLLM / Token Throughput

• VLM Serving in Production: Stabilizing Image & Video Inference with Vision Token Budgets

  A systematic guide to production VLM serving: image/video input governance, vision token budgets, preprocessing queues, cost control, and a go-live checklist for stable multimodal inference.

  07/02/2026 VLM / LLM Serving / Multimodal AI / Vision Token Budget / Multimodal Inference / Inference / Vision Token / vLLM / SGLang

• Multi-LoRA Production Practices: Stable Multi-Tenant Inference with Adapter Caching and Hot/Cold Tiering

  A systematic guide to Multi-LoRA serving in production: adapter caching, hot/cold tiering, rank-aware scheduling, eviction strategies, and deployment checks to stably serve hundreds of custom LoRA adapters under GPU memory constraints.

  07/01/2026 LLM Serving / LoRA / Inference / GPU / Multi Tenant / Adapter Cache / vLLM

• Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

  A systematic guide on how Continuous Batching improves LLM inference throughput via iteration-level scheduling, covering static batching pain points, scheduling mechanics, KV Cache constraints, key parameters, monitoring metrics, and deployment pitfalls.

  07/01/2026 LLM Serving / Continuous Batching / Inference Optimization / GPU / vLLM

• KV Cache Quantization in Production: Reducing Long-Context Inference Memory with FP8/INT8

  A systematic guide to KV cache quantization for long-context LLM inference: value, principles, vLLM/TensorRT-LLM implementation, evaluation metrics, and production risks to help teams safely reduce memory and latency costs.

  07/01/2026 LLM / Inference Optimization / KV Cache / Quantization / LLMOps / vLLM / TensorRT-LLM

• Continuous Batching in Production: Keeping LLM Services GPU-Saturated Under High Concurrency

  A systematic guide to continuous batching for LLM serving: core principles, scheduling strategies, KV Cache constraints, engineering deployment steps, and monitoring metrics. Covers the full path from proof-of-concept to production, helping teams boost throughput while controlling tail latency in high-concurrency scenarios.

  07/01/2026 LLM Serving / Continuous Batching / vLLM / KV Cache / Performance Engineering / Inference Optimization / GPU Scheduling

• Speculative Decoding in Production: Reducing LLM Per-User Latency with EAGLE and Draft Models

  A practical guide to Speculative Decoding for LLM inference: draft generation, parallel verification, acceptance length, and production metrics. Compares Draft Model, Medusa, and EAGLE with rollout and rollback checklists.

  06/30/2026 LLM Inference / Speculative Decoding / vLLM / EAGLE

• KV Cache Reuse in Practice: Prefix Caching, Offloading, and Long-Context Cost Management

  A systematic guide to KV Cache reuse for long-context LLM serving: Prefix Caching, RadixAttention, multi-tier offloading, hit rate, latency, cost, and production checklists.

  06/29/2026 LLM Inference / KV Cache / Prefix Caching / vLLM / SGLang

• Speculative Decoding in Production: Why Low QPS Speeds Up but High Load Slows Down

  A production-oriented analysis of speculative decoding's benefit boundaries, covering draft models, verification, acceptance rate, request load, and batching, with a pre-deployment checklist.

  06/29/2026 LLM Inference / Speculative Decoding / vLLM / TensorRT-LLM / Performance Optimization

• LLM Quantization in Practice: FP8, INT4, AWQ, and Production Inference Trade-offs

  A systematic guide to LLM inference quantization: FP8, INT4, AWQ, GPTQ, KV Cache quantization, benefits, risks, selection methods, and a go-live checklist for production engineers.

  06/29/2026 LLM Inference / Quantization / vLLM / TensorRT-LLM / Model Serving

• PagedAttention in Action: Why KV Cache Scheduling Matters More Than Compute for LLM Serving

  A deep dive into KV Cache memory fragmentation, PagedAttention, continuous batching, and prefix caching — and how they boost throughput and reduce latency in LLM inference.

  06/29/2026 LLM Inference / PagedAttention / KV Cache / vLLM / LLM Engineering