Tag: Inference Infrastructure

2 articles

• LLM LoRA Adapter Cache in Production: Reducing Dynamic Loading Jitter with Hot/Cold Tiering and Eviction Strategies

  This article covers production governance for LoRA adapter caches in multi-tenant LLM serving, including dynamic loading, hot/cold tiering, eviction strategies, tenant quotas, version releases, and go-live checks to reduce first-token latency and GPU memory jitter.

  07/05/2026 LLM Serving / LoRA / Adapter Cache / Inference Infrastructure / Hot/Cold Tiering / Multi-Tenant / Dynamic Loading / Eviction Strategy

• LLM Serving Overload Protection: SLO-Driven Autoscaling and Admission Control

  A practical guide to building overload protection for LLM inference services using SLO-driven autoscaling and admission control. Covers queue length, token pressure, TTFT/TPOT, and tenant quotas to prevent system collapse under traffic spikes.

  07/01/2026 LLM Serving / Autoscaling / SLO / Admission Control / Overload Protection / Inference Infrastructure