Tag: LLM Inference

6 articles

• Edge LLM Inference in Production: Running Local Models Reliably with GGUF, mmap, and CPU/GPU Offloading

  A systematic guide to deploying open-weight LLMs on local workstations, edge nodes, and internal servers. Covers GGUF quantization selection, mmap memory-mapped loading, CPU/GPU layer offloading tuning, service governance, and a production launch checklist to elevate edge inference from 'it runs' to 'operationally sound'.

  07/06/2026 LLM Inference / GGUF / Edge AI / llama.cpp / Model Deployment / Quantization / Local Inference

• Speculative Decoding in Production: Reducing LLM Per-User Latency with EAGLE and Draft Models

  A practical guide to Speculative Decoding for LLM inference: draft generation, parallel verification, acceptance length, and production metrics. Compares Draft Model, Medusa, and EAGLE with rollout and rollback checklists.

  06/30/2026 LLM Inference / Speculative Decoding / vLLM / EAGLE

• KV Cache Reuse in Practice: Prefix Caching, Offloading, and Long-Context Cost Management

  A systematic guide to KV Cache reuse for long-context LLM serving: Prefix Caching, RadixAttention, multi-tier offloading, hit rate, latency, cost, and production checklists.

  06/29/2026 LLM Inference / KV Cache / Prefix Caching / vLLM / SGLang

• Speculative Decoding in Production: Why Low QPS Speeds Up but High Load Slows Down

  A production-oriented analysis of speculative decoding's benefit boundaries, covering draft models, verification, acceptance rate, request load, and batching, with a pre-deployment checklist.

  06/29/2026 LLM Inference / Speculative Decoding / vLLM / TensorRT-LLM / Performance Optimization

• LLM Quantization in Practice: FP8, INT4, AWQ, and Production Inference Trade-offs

  A systematic guide to LLM inference quantization: FP8, INT4, AWQ, GPTQ, KV Cache quantization, benefits, risks, selection methods, and a go-live checklist for production engineers.

  06/29/2026 LLM Inference / Quantization / vLLM / TensorRT-LLM / Model Serving

• PagedAttention in Action: Why KV Cache Scheduling Matters More Than Compute for LLM Serving

  A deep dive into KV Cache memory fragmentation, PagedAttention, continuous batching, and prefix caching — and how they boost throughput and reduce latency in LLM inference.

  06/29/2026 LLM Inference / PagedAttention / KV Cache / vLLM / LLM Engineering