Tag: model-deployment

2 articles

• LLM Weight Quantization in Production: Preventing Quality Regression with Calibration Sets, FP8/W4A16 Dual-Track Benchmarks, and Quality Gates

  LLM weight quantization is more than just reduced VRAM. This guide covers calibration set selection, FP8/W4A16 dual-track benchmarking, hardware matching, quality gates, and canary rollouts using vLLM, LLM Compressor, and TensorRT-LLM.

  08/31/2026 llm-serving / quantization / fp8 / awq / model-deployment

• Edge LLM Inference in Production: Running Local Models Reliably with GGUF, mmap, and CPU/GPU Offloading

  A systematic guide to deploying open-weight LLMs on local workstations, edge nodes, and internal servers. Covers GGUF quantization selection, mmap memory-mapped loading, CPU/GPU layer offloading tuning, service governance, and a production launch checklist to elevate edge inference from 'it runs' to 'operationally sound'.

  07/06/2026 LLM Inference / GGUF / Edge AI / llama.cpp / Model Deployment / Quantization / Local Inference