Tag: Inference Infrastructure

3 articles

• LLM GPU Multi-Tenant Isolation in Production: Sharing Inference Stably with MIG Profiles, Time-Slicing Layering, and Reconfiguration Gates

  A systematic guide to MIG, Time-Slicing, and MPS isolation differences for LLM inference on shared GPUs. Covers profile selection, Kubernetes scheduling, reconfiguration gates, monitoring, and rollback to build a verifiable production plan balancing utilization, latency, and fault boundaries.

  07/12/2026 LLM Serving / NVIDIA MIG / GPU Multi-Tenancy / Kubernetes / Inference Infrastructure

• LLM LoRA Adapter Cache in Production: Reducing Dynamic Loading Jitter with Hot/Cold Tiering and Eviction Strategies

  This article covers production governance for LoRA adapter caches in multi-tenant LLM serving, including dynamic loading, hot/cold tiering, eviction strategies, tenant quotas, version releases, and go-live checks to reduce first-token latency and GPU memory jitter.

  07/05/2026 LLM Serving / LoRA / Adapter Cache / Inference Infrastructure / Hot/Cold Tiering / Multi-Tenant / Dynamic Loading / Eviction Strategy

• LLM Serving Overload Protection: SLO-Driven Autoscaling and Admission Control

  A practical guide to building overload protection for LLM inference services using SLO-driven autoscaling and admission control. Covers queue length, token pressure, TTFT/TPOT, and tenant quotas to prevent system collapse under traffic spikes.

  07/01/2026 LLM Serving / Autoscaling / SLO / Admission Control / Overload Protection / Inference Infrastructure