LLM Quantization in Practice: FP8, INT4, AWQ, and Production Inference Trade-offs

Background: Quantization Solves More Than Just “Making the Model Smaller”

After deploying a large model, the most underestimated costs are not the model file itself, but the persistent consumption of VRAM, bandwidth, compute kernels, and KV Cache during inference. Just because a model is trained in BF16 or FP16 doesn’t mean it must run at the same precision in production. Many production systems use quantization to convert weights, activations, or KV Cache from high-precision to low-precision representations, reducing VRAM usage, increasing throughput, or enabling larger models to run on fewer devices.

But quantization is not as simple as “compressing the parameters.” The vLLM documentation describes quantization as trading precision for smaller memory footprint, allowing larger models to run on more devices, and lists formats like AWQ, GPTQ, bitsandbytes, FP8, INT4, INT8, and quantized KV Cache. TensorRT-LLM also classifies quantization as a core inference optimization capability, covering recipes like FP4, FP8, FP8 KV Cache, W4A16 GPTQ, W4A8 GPTQ, W4A16 AWQ, and W4A8 AWQ. In other words, production quantization is no longer a single algorithm problem; it’s a combined engineering effort involving model format, calibration method, inference engine, hardware kernels, and business quality evaluation.

This article doesn’t debate “which quantization method is theoretically the best.” Instead, it explains from an engineering deployment perspective: when to use FP8, when to use INT4/AWQ, why KV Cache should also be part of the quantization decision, and what checks are necessary before going live.

Core Concepts: Weights, Activations, and KV Cache Are Three Different Objects

Before quantizing, clarify what is being quantized. Different objects have different impacts and risks.

Weight Quantization: The Most Intuitive VRAM Benefit

Weight quantization is the most common entry point. For example, converting weights from FP16/BF16 to INT8, INT4, or other formats. Its most direct benefit is a smaller model weight size and lower VRAM load. For VRAM-constrained deployment environments, weight quantization often determines whether a model can fit on a single GPU.

However, weight quantization does not automatically lead to end-to-end speedup. If the inference framework lacks corresponding high-performance kernels, or if the quantized format requires frequent dequantization at runtime, the model file may be smaller, but actual latency and throughput may not improve. Therefore, selection cannot rely solely on labels like “4-bit”; you must check whether the serving framework natively supports that format.

Activation Quantization: Higher Reward, Higher Risk

Activation quantization affects runtime intermediate results. It can further reduce bandwidth and compute costs but is more sensitive to numerical stability. NVIDIA’s PTQ documentation emphasizes that quantization performance gains depend on the quantization range, the difference between original and target precision, and the algorithm used; calibration determines scaling factors using representative input data. A common issue with activations is outliers — a few abnormally large values can widen the quantization range, degrading effective precision for most normal values.

This is why methods like SmoothQuant and AWQ exist. They don’t simply compress all numbers to low bits; they reduce error through scaling, calibration, or protecting critical channels.

KV Cache Quantization: Essential for Long-Context Serving

Many teams initially focus only on model weights and overlook KV Cache. In long-context, multi-turn dialogue, and high-concurrency services, KV Cache can become the primary source of VRAM growth. TensorRT-LLM already explicitly provides FP8 KV Cache and NVFP4 KV Cache options, indicating that KV Cache precision has become part of inference configuration.

The benefit of KV Cache quantization typically manifests in long-context throughput and concurrency, but the risk is also clear: if cache precision is too low, attention quality in the latter half of long texts may degrade, manifesting as citation errors, detail loss, unstable code completion, or deteriorating multi-turn dialogue memory. Therefore, KV Cache quantization cannot be evaluated with short prompt benchmarks alone; it must include long-context and multi-turn examples.

Main Approaches: When to Use FP8, INT4, AWQ, and GPTQ

FP8: A Robust Choice for Data Center GPUs

FP8’s advantage is its alignment with modern GPU hardware optimization. Compared to INT4, it typically retains better dynamic range; compared to FP16/BF16, it reduces bandwidth and storage pressure. TensorRT-LLM documentation lists recipes like FP8 Per Tensor, FP8 Block Scaling, FP8 Rowwise, and FP8 KV Cache, and notes that the default PyTorch backend supports FP4 and FP8 quantization on newer Hopper and Blackwell GPUs.

If your goal is production serving, relatively abundant GPU resources, and business quality stability as a priority, FP8 is often easier to deploy than extreme INT4. Especially on devices like H100/H200/B200, FP8’s toolchain and kernel support are increasingly mature.

Use Cases:

• Online inference services requiring stable throughput and low quality risk
• Large models, but not yet VRAM-constrained enough to require INT4
• Using modern inference frameworks like TensorRT-LLM, vLLM, SGLang
• Teams capable of business regression testing but preferring simpler quantization strategies

INT4: Strong Compression When VRAM is Tight

INT4’s appeal is straightforward: high compression ratio. vLLM’s INT4 W4A16 documentation states that INT4 weight quantization can save memory and accelerate inference, especially for reducing model size and maintaining low latency in low-QPS workloads. Hugging Face’s quantization selection guide also classifies GPTQ and AWQ as calibration-required methods, noting that AWQ typically achieves higher accuracy at 4-bit, and GPTQ is also commonly used for high-accuracy quantization.

However, INT4 is not “faster by default.” It depends on hardware, kernels, batch size, sequence length, and serving framework. Low-QPS single requests may benefit from smaller VRAM and lower memory bandwidth; high-concurrency scenarios depend on kernel maturity.

Use Cases:

• Single GPU VRAM is tight, needing to deploy a larger model
• Edge devices, local inference, cost-sensitive services
• Acceptable to perform heavier calibration and regression testing
• Inference framework has native high-performance support for the target INT4 format

AWQ: Protecting a Few Critical Channels Instead of Averaging All Weights

AWQ (Activation-aware Weight Quantization) ‘s core idea is that not all weights are equally important. The AWQ paper shows that protecting about 1% of salient weights can significantly reduce quantization error, and identifying critical channels should reference activation distributions, not just weights alone. It applies equivalent scaling to salient channels, avoiding inefficient hardware mixed precision while reducing quantization error.

This is valuable for engineering deployment. The risk of traditional “uniform layer quantization” is that if certain critical channels are damaged, the model may exhibit hidden degradation on tasks like math, code, and complex instructions. AWQ’s approach is closer to production needs: use a small amount of calibration data to identify sensitive parts, maintain hardware friendliness, and minimize quality loss.

Use Cases:

• Need 4-bit weight quantization but don’t want significant quality degradation
• Have representative calibration data
• Model includes code, math, reasoning tasks that cannot be evaluated with generic chit-chat examples
• Serving framework supports AWQ checkpoints and corresponding kernels

GPTQ: Mature, but Beware of Calibration Set Overfitting

GPTQ is another common post-training quantization approach. Hugging Face documentation classifies GPTQ/GPTQModel as calibration-based quantization, noting it requires calibration data and a separate calibration step, with the advantage of typically higher accuracy and potential inference speedup.

GPTQ’s risk lies in calibration data selection. If the calibration set is too narrow, the model may appear normal on test sets but degrade on real business inputs. For vertical domains like customer service, code, finance, insurance, and law, you cannot use only general English corpora for calibration; at minimum, include de-identified samples of real business prompts.

Quick Comparison of Quantization Approaches

Method	Typical Precision	VRAM Benefit	Quality Risk	Calibration Need	Best Scenario
FP8	W8A8	Medium	Low	Light	Data center GPU online serving
INT8	W8A16	Medium	Low	Light	General inference, conservative first choice
INT4	W4A16	High	Medium-High	Required	Extremely VRAM-constrained
AWQ	W4A16	High	Medium	Required	Pursuing quality in 4-bit
GPTQ	W4A16	High	Medium	Required	Mature solution, watch calibration generalization
FP8 KV Cache	KV FP8	Grows with context	Medium	Light	Long-context high concurrency

A Practical Selection Framework

Production environments can follow this sequence for decision-making.

Step 1: Identify the Bottleneck Type

• If the bottleneck is model doesn’t fit in VRAM, prioritize weight quantization, e.g., INT8, INT4, AWQ, or GPTQ.
• If the bottleneck is throughput and GPU utilization, first check inference framework, batching, kernels, FP8, PagedAttention, continuous batching — not just model file size.
• If the bottleneck is long-context concurrency, KV Cache quantization, prefix cache, paged cache, and chunked prefill must all be included in the same test suite.

Step 2: Start with Conservative Approaches

A reasonable production path is usually:

BF16/FP16 baseline → FP8 weight/activation or INT8
    → INT4 AWQ/GPTQ → KV Cache FP8/NVFP4
    → Mixed precision and layer-wise sensitivity optimization

Don’t aim for 2-bit, binary, or extreme low precision from the start. The OpenPangu quantization empirical study (revised June 26, 2026) evaluated RTN, GPTQ, AWQ, SmoothQuant, GPTAQ, BiLLM, and SliM-LLM on Ascend 910B1 NPUs. The conclusion shows that 8-bit weight-only is essentially lossless for OpenPangu 1B/7B, 4-bit is still practical for 7B but causes more noticeable damage to reasoning, math, and code tasks for 1B; most 2-bit and binarization settings approach random performance. Such results remind us: low bit does not mean free gains; model scale and task type significantly affect outcomes.

Step 3: Let the Inference Framework Determine Available Formats

Quantization format must be tied to runtime. The support matrix for various frameworks is as follows:

Framework	Supported Quantization Formats
vLLM	AWQ, GPTQ, bitsandbytes, FP8, INT4, INT8, GGUF, TorchAO, quantized KV Cache
TensorRT-LLM	FP4, FP8, AWQ, GPTQ, KV Cache quantization (including NVFP4)
Hugging Face Transformers	Model loading, offline conversion, experimentation, and partial online inference

Therefore, don’t just ask “which quantization algorithm is best.” Instead, ask:

• Is my GPU/CPU/NPU supported?
• Is the inference engine natively supported?
• Are there high-performance kernels?
• Is the current model architecture supported?
• Are tensor parallelism, pipeline parallelism, LoRA, tool calling, and long context supported?
• Can it be exported and rolled back?

Engineering Implementation: An Executable Process

1. Establish an FP16/BF16 Baseline

Before quantizing, fix the baseline, including:

• Model version, tokenizer version, inference framework version
• Generation parameters like temperature, top_p, max_tokens
• Business evaluation dataset
• Latency, throughput, VRAM, time-to-first-token, tokens-per-second
• Long-context and multi-turn dialogue examples

Without a baseline, there is no quantization benefit evaluation. Many disputes about “quantization made things worse” actually stem from inconsistent test conditions.

2. Prepare Calibration and Evaluation Sets

The calibration set determines quantization parameters; the evaluation set judges business quality. They must not be identical.

It is recommended that the calibration set includes:

• De-identified samples of real business prompts
• Short text, long text, multi-turn dialogue
• High-sensitivity tasks like code, math, tables, JSON output
• Mixed Chinese and English input
• High-risk boundary examples

The evaluation set should cover metrics like accuracy, format stability, refusal boundaries, hallucination rate, tool call parameters, and structured output validity.

3. First Run Small Models or Small Traffic Canary

Don’t directly replace the production model with a quantized model at full scale. A safer approach is:

Offline evaluation → Shadow traffic → Small percentage canary → Gradual rollout by business → Full switch

In the shadow traffic phase, don’t return results to users; only record output differences between the quantized model and the original model. In the small percentage canary phase, monitor complaint rate, retry rate, timeout rate, format error rate, and human fallback ratio in real time.

4. Manage the Quantized Model as an Independent Model Version

A quantized checkpoint should not just be an accessory file of the original model. It should have an independent version number, quantization configuration, calibration data summary, evaluation report, and rollback strategy.

Example metadata to record:

base_model: Qwen/Qwen3-xxB
quant_method: AWQ
weight_dtype: int4
activation_dtype: fp16
kv_cache_dtype: fp8
calibration_dataset: business-sampled-2026-06
inference_engine: vLLM x.y.z
hardware: H100 / A100 / 4090 / Ascend 910B
quality_report: report-20260629.md
rollback_model: bf16-baseline

Common Misconceptions

Misconception 1: Larger Models Are More Suitable for Extremely Low Bits

Not necessarily. Larger models are sometimes more robust to 4-bit, but this does not mean any large model is suitable for 2-bit or binary. The latest OpenPangu NPU empirical study also shows that extremely low precision remains difficult, with most 2-bit and binarization settings experiencing severe degradation.

Misconception 2: Quantization Only Affects Accuracy, Not System Behavior

Incorrect. Quantization affects latency distribution, throughput, VRAM peaks, kernel selection, batching strategy, long-context stability, and output format validity. For systems requiring JSON, tool calls, or code generation, even slight numerical perturbations can amplify into format errors.

Misconception 3: If Benchmark Scores Are Close, It’s Ready for Production

General benchmarks only tell part of the story. Real business cares more about long-tail inputs, fixed formats, factual recall, domain terminology, refusal boundaries, and context consistency. Quantized models must pass business regression before going live.

Misconception 4: Pre-quantized Models Are Always More Reliable Than Self-quantized Ones

Pre-quantized models save time, but you don’t know if their calibration data and target workload match. They are suitable for quick experiments but do not exempt you from production evaluation.

Go-Live Checklist

Quality Checks

• Compare item-by-item with FP16/BF16 baseline
• Check accuracy on core business questions
• Check code, math, tables, JSON output
• Check multi-turn dialogue and long context
• Check refusal, safety, and compliance boundaries

Performance Checks

• Time-to-first-token
• Average tokens per second
• P95/P99 latency
• Maximum concurrency per GPU
• VRAM peak
• Throughput curve under different batch sizes

Runtime Checks

• Is the inference framework natively supported for this format?
• Are high-performance kernels used, not fallback?
• Are tensor parallel / pipeline parallel working correctly?
• Are LoRA, tool calling, and structured output compatible?
• Does KV Cache precision affect long context?
• Is fast rollback to baseline supported?

Conclusion: Quantization is Systems Engineering, Not a Compression Button

The value of LLM quantization is clear: reduce VRAM, increase throughput, and enable larger models in lower-cost deployment environments. But in production, the real question is not “whether to quantize,” but what to quantize, to what precision, on which hardware, executed by which inference framework, and validated with what business data.

A robust practical principle: first establish an FP16/BF16 baseline, start with conservative approaches like FP8 or INT8, confirm benefits, then move to INT4/AWQ/GPTQ; if the service includes long context and high concurrency, separately evaluate KV Cache quantization. Quantized models must have independent versions, evaluation reports, and rollback paths. Only then is quantization a production optimization, not an uncontrollable compression experiment.

Key References

1. vLLM Quantization Documentation
2. vLLM INT4 W4A16 Documentation
3. NVIDIA TensorRT-LLM Quantization Documentation
4. Hugging Face Transformers Quantization Documentation
5. Hugging Face Quantization Method Selection Guide
6. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
7. NVIDIA Technical Blog: Optimizing LLMs for Performance and Accuracy with Post-Training Quantization
8. An Empirical Study of OpenPangu Quantization on Ascend NPUs