Tag: Long Context

4 articles

• Attention Sinks in Production: Using StreamingLLM for Long Session Generation

  Explains the Attention Sink phenomenon, how StreamingLLM retains initial sink tokens and a recent window to reduce GPU memory pressure without fine-tuning, and provides production deployment boundaries, monitoring metrics, and a go-live checklist.

  06/30/2026 LLM Serving / StreamingLLM / KV Cache / Long Context

• KV Cache Quantization in Practice: Managing Memory and Throughput for Long-Context Inference

  KV Cache quantization isn't just about lowering precision; it's a complete serving governance strategy involving memory capacity, attention bandwidth, model accuracy, calibration data, and online fallback policies.

  06/30/2026 LLM Serving / KV Cache / Quantization / Long Context

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift LLM inference bottlenecks from single-card throughput to tail-latency governance. This post dives into Prefill-Decode disaggregation, KV cache transfer, resource planning, and a pre-launch checklist to move LLM serving from 'working' to stable and scalable.

  06/29/2026 LLM Serving / Prefill Decode / KV Cache / Inference Optimization / Long Context / Tail Latency / RAG / Agent

• Prefill-Decode Disaggregation in Practice: Taming Latency for Long-Context LLM Serving

  Long contexts and agentic requests shift the LLM inference bottleneck from single-GPU throughput to tail-latency management. This article dives into the principles, trade-offs, KV cache transfer, resource planning, and a go-live checklist for Prefill-Decode disaggregation.

  06/29/2026 LLM Serving / Prefill-Decode / KV Cache / Inference Optimization / Long Context / Latency Management