Tag: Distributed Systems

2 articles

• LLM Context Parallel in Production: Breaking Long-Context Bottlenecks with Sequence Sharding, Ring/Ulysses Communication, and Causal Load Balancing

  An in-depth guide to Context Parallel for long-context LLM training, covering Ring Attention vs. Ulysses communication, causal load balancing strategies, and production deployment checks to scale context from 8K to 128K+ tokens on multi-GPU clusters.

  07/16/2026 LLM Training / Context Parallel / Distributed Systems / Long Context / Ring Attention

• LLM MoE Expert Parallel in Production: Stabilizing Inference Throughput with Expert Load Profiling, All-to-All Budgeting, and Dynamic Replicas

  MoE's sparse activation doesn't guarantee efficient inference. This article breaks down Expert Parallel's All-to-All communication, expert hotspots, and dynamic replica mechanisms, offering a complete production governance strategy from load profiling to topology placement and capacity gating.

  07/11/2026 LLM Serving / Mixture of Experts / Expert Parallel / GPU Inference / Distributed Systems