Tag: Quantization

4 articles

• Edge LLM Inference in Production: Running Local Models Reliably with GGUF, mmap, and CPU/GPU Offloading

  A systematic guide to deploying open-weight LLMs on local workstations, edge nodes, and internal servers. Covers GGUF quantization selection, mmap memory-mapped loading, CPU/GPU layer offloading tuning, service governance, and a production launch checklist to elevate edge inference from 'it runs' to 'operationally sound'.

  07/06/2026 LLM Inference / GGUF / Edge AI / llama.cpp / Model Deployment / Quantization / Local Inference

• KV Cache Quantization in Production: Reducing Long-Context Inference Memory with FP8/INT8

  A systematic guide to KV cache quantization for long-context LLM inference: value, principles, vLLM/TensorRT-LLM implementation, evaluation metrics, and production risks to help teams safely reduce memory and latency costs.

  07/01/2026 LLM / Inference Optimization / KV Cache / Quantization / LLMOps / vLLM / TensorRT-LLM

• KV Cache Quantization in Practice: Managing Memory and Throughput for Long-Context Inference

  KV Cache quantization isn't just about lowering precision; it's a complete serving governance strategy involving memory capacity, attention bandwidth, model accuracy, calibration data, and online fallback policies.

  06/30/2026 LLM Serving / KV Cache / Quantization / Long Context

• LLM Quantization in Practice: FP8, INT4, AWQ, and Production Inference Trade-offs

  A systematic guide to LLM inference quantization: FP8, INT4, AWQ, GPTQ, KV Cache quantization, benefits, risks, selection methods, and a go-live checklist for production engineers.

  06/29/2026 LLM Inference / Quantization / vLLM / TensorRT-LLM / Model Serving