Tag: inference

3 articles

• VLM Serving in Production: Stabilizing Image & Video Inference with Vision Token Budgets

  A systematic guide to production VLM serving: image/video input governance, vision token budgets, preprocessing queues, cost control, and a go-live checklist for stable multimodal inference.

  07/02/2026 VLM / LLM Serving / Multimodal AI / Vision Token Budget / Multimodal Inference / Inference / Vision Token / vLLM / SGLang

• Multi-LoRA Production Practices: Stable Multi-Tenant Inference with Adapter Caching and Hot/Cold Tiering

  A systematic guide to Multi-LoRA serving in production: adapter caching, hot/cold tiering, rank-aware scheduling, eviction strategies, and deployment checks to stably serve hundreds of custom LoRA adapters under GPU memory constraints.

  07/01/2026 LLM Serving / LoRA / Inference / GPU / Multi Tenant / Adapter Cache / vLLM

• PagedAttention in Production: Boosting LLM Serving Throughput with Paged KV Cache

  A deep dive into how PagedAttention mitigates GPU memory fragmentation and over-reservation in LLM inference via paged KV cache management, helping engineering teams increase throughput, reduce latency variance, and providing a production deployment checklist and monitoring strategy.

  06/30/2026 llm-serving / pagedattention / kv-cache / inference