Tag: GPU

4 articles

• LLM Distributed Inference Communication Fault Debugging: Locating Collective Communication Hangs with NCCL RAS, Flight Recorder, and Topology Baselines

  When NCCL collective communication hangs in large-scale distributed LLM inference, you're often left with only a vague timeout log. This article explains how to use RAS, Flight Recorder, topology baselines, and anomalous rank detection to build observable, rollback-capable communication gating for production deployments.

  07/13/2026 LLM Serving / NCCL / Distributed Inference / GPU / Reliability

• LLM LoRA Adapter Cache in Production: Stabilizing Multi-Tenant Fine-Tuning Services with Heat Tiering, Cold-Start Budgets, and Eviction Policies

  Production strategies for LoRA adapter cache management in multi-tenant fine-tuning services: heat tiering, cold-start budgets, eviction policies, and a governance framework for stable model serving.

  07/09/2026 LLM Serving / LoRA / Adapter Cache / Multi Tenant / Model Serving / GPU / Fine Tuning / RAG / vLLM

• Multi-LoRA Production Practices: Stable Multi-Tenant Inference with Adapter Caching and Hot/Cold Tiering

  A systematic guide to Multi-LoRA serving in production: adapter caching, hot/cold tiering, rank-aware scheduling, eviction strategies, and deployment checks to stably serve hundreds of custom LoRA adapters under GPU memory constraints.

  07/01/2026 LLM Serving / LoRA / Inference / GPU / Multi Tenant / Adapter Cache / vLLM

• Continuous Batching in Production: Boosting LLM Serving Throughput with Dynamic Batching

  A systematic guide on how Continuous Batching improves LLM inference throughput via iteration-level scheduling, covering static batching pain points, scheduling mechanics, KV Cache constraints, key parameters, monitoring metrics, and deployment pitfalls.

  07/01/2026 LLM Serving / Continuous Batching / Inference Optimization / GPU / vLLM