Tag: Long Context

6 articles

• LLM Context Parallel in Production: Breaking Long-Context Bottlenecks with Sequence Sharding, Ring/Ulysses Communication, and Causal Load Balancing

  An in-depth guide to Context Parallel for long-context LLM training, covering Ring Attention vs. Ulysses communication, causal load balancing strategies, and production deployment checks to scale context from 8K to 128K+ tokens on multi-GPU clusters.

  07/16/2026 LLM Training / Context Parallel / Distributed Systems / Long Context / Ring Attention

• LLM RoPE Scaling in Production: Preventing Long-Context Degradation with Config Fingerprints, Length Buckets, and Dual-Track Evaluation

  A systematic guide to safely deploying long-context LLMs with RoPE scaling. Covers config fingerprints, length buckets, dual-track evaluation, and graceful rollback to prevent context degradation in production.

  07/13/2026 LLM Inference / Long Context / RoPE / Model Evaluation / Production Engineering / Positional Encoding

• Attention Sinks in Production: Using StreamingLLM for Long Session Generation

  Explains the Attention Sink phenomenon, how StreamingLLM retains initial sink tokens and a recent window to reduce GPU memory pressure without fine-tuning, and provides production deployment boundaries, monitoring metrics, and a go-live checklist.

  06/30/2026 LLM Serving / StreamingLLM / KV Cache / Long Context

• KV Cache Quantization in Practice: Managing Memory and Throughput for Long-Context Inference

  KV Cache quantization isn't just about lowering precision; it's a complete serving governance strategy involving memory capacity, attention bandwidth, model accuracy, calibration data, and online fallback policies.

  06/30/2026 LLM Serving / KV Cache / Quantization / Long Context

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift LLM inference bottlenecks from single-card throughput to tail-latency governance. This post dives into Prefill-Decode disaggregation, KV cache transfer, resource planning, and a pre-launch checklist to move LLM serving from 'working' to stable and scalable.

  06/29/2026 LLM Serving / Prefill Decode / KV Cache / Inference Optimization / Long Context / Tail Latency / RAG / Agent

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift the LLM inference bottleneck from single-GPU throughput to tail-latency management. This article dives into the principles, trade-offs, KV cache transfer, resource planning, and a go-live checklist for Prefill-Decode disaggregation.

  06/29/2026 LLM Serving / Prefill-Decode / KV Cache / Inference Optimization / Long Context / Latency Management