Can I use generic public datasets for quantization calibration?

They can serve as a starting point, but production models should use data that closely matches real business input distributions. Domain, language, length, and input template mismatches all affect calibration results, especially for AWQ, GPTQ, and static activation quantization.

Why keep the KV Cache configuration unchanged during quantization rollout?

To isolate variables. If you change weight quantization and KV Cache precision simultaneously, quality, VRAM, and latency changes become hard to attribute. This workflow covers only weight or weight-plus-activation quantization; KV Cache should be validated in separate experiments.

LLM Weight Quantization in Production: Preventing Quality Regression with Calibration Sets, FP8/W4A16 Dual-Track Benchmarks, and Quality Gates

Q: Should I choose between FP8 and W4A16?

Don't decide based on the format alone. Build separate FP8 and W4A16 candidate baselines on the same hardware, inference engine, and request set, then compare task quality, VRAM, throughput, and tail latency side by side.

Once an LLM is in production, quantization is often treated as a simple deployment optimization: swap BF16/FP16 for FP8, INT8, or INT4, watch VRAM shrink and theoretical throughput rise, then ship it.

The problem is that quantization changes the numerical representation of the model. Even if the model loads cleanly and the API returns sensible output, that doesn’t mean business quality is preserved. Meanwhile, whether a given quantization format actually delivers speedups depends on GPU architecture, the inference engine, quantization kernels, batch shapes, and model structure.

So production quantization decisions shouldn’t be framed as:

“W4A16 is smaller than FP16, so use W4A16.”

They should be framed as:

“On the specified hardware and request distribution, the candidate quantized artifact passes the quality gate and delivers verifiable gains on at least one key objective — VRAM, throughput, or latency — so it is approved for canary release.”

This article focuses exclusively on model weights, or weight-plus-activation quantization. To avoid conflating this with KV Cache quantization, all experiments below keep KV Cache precision, scheduling policy, and serving parameters fixed by default.

1. Core Principles: Decompose Quantization into Three Independent Problems

1.1 The Quantization Format Determines “What Gets Compressed”

Common production schemes fall into two broad categories:

Weight-only: e.g., W4A16. Weights are compressed to 4 bits while activations retain higher precision. This primarily addresses model VRAM footprint and suits low-QPS, VRAM-constrained, or older-GPU deployments.
Weight + Activation: e.g., FP8 W8A8. Both weights and activations enter a low-precision compute path, relying more on native hardware support and corresponding kernels, but with better chances of compute-throughput gains on suitable platforms.

vLLM’s current quantization support matrix includes AWQ, GPTQ, Marlin, LLM Compressor FP8/INT8, bitsandbytes, GGUF, and more, with explicit per-GPU-architecture support notes. TensorRT-LLM likewise treats FP8, FP4, and W4A16 AWQ/GPTQ as distinct quantization recipes rather than a single quantized=true switch.

The first principle of production design is therefore: quantization format, quantization algorithm, and runtime kernels must be chosen as a single integrated unit.

1.2 The Calibration Set Determines “What Distribution the Model Saw Before Being Compressed”

AWQ, GPTQ, SmoothQuant, and some static activation quantization methods require calibration data. LLM Compressor’s official documentation explicitly recommends that calibration sets match the target business distribution and notes that many calibration algorithms can start iterating with relatively small datasets.

The key point isn’t “more samples is better” — it’s representativeness:

If production traffic is primarily Chinese insurance-clause Q&A, but the calibration set is all English Wikipedia;
If production requests typically have 8K context, but calibration only uses short inputs of a few hundred tokens;
If the model is a code assistant, but the calibration set contains no code at all;

Then the activation distribution the quantizer sees may diverge significantly from production reality.

A more practical calibration set should cover:

Dimension	Description
Language & Domain	Primary languages, primary business domains
Templates	Real prompts / chat templates
Length	Buckets for short, medium, and long inputs
Tasks	High-frequency tasks and critical long-tail tasks
Structured Inputs	Code, tables, JSON, etc.
High-Value Types	Key requests sensitive to quality

The calibration set is not an eval set. Calibration determines quantization parameters; eval proves there’s no unacceptable regression after quantization. The two must be strictly separated.

1.3 Runtime and Hardware Determine “Whether Compression Actually Makes It Faster”

The same W4A16 checkpoint can yield completely different gains on different GPUs, kernels, and batch sizes. You can’t equate a smaller checkpoint file with faster inference. Production evaluation must answer:

Does the current GPU natively support the corresponding low-precision compute?
Is the serving engine actually invoking the target quantization kernel?
Is performance consistent across small and large batches?
Can VRAM savings translate into higher concurrency or fewer replicas?
Which of TTFT, TPOT, and throughput is the real business bottleneck?

2. Engineering Execution: Establish Dense, FP8, and W4A16 Baselines

Don’t tweak parameters in one environment and “feel” that things got faster. A more robust approach is to fix the experimental matrix.

Baseline A: Dense Baseline

Use the current production BF16/FP16 artifact with these fixed:

Model version;
Tokenizer / chat template;
Serving engine version;
Tensor Parallel / Pipeline Parallel;
Maximum context length;
KV Cache precision;
Batch and concurrency parameters;
GPU model and count.

The Dense baseline isn’t about achieving peak performance — it’s about providing the single comparable reference point for quality and performance.

Candidate B: FP8

FP8 is a reasonable first candidate on modern GPUs. TensorRT-LLM’s FP8 guide explicitly states that building an FP8 engine from an FP16/BF16 checkpoint allows calibration configuration, and emphasizes that output quality must be re-validated after quantization.

The FP8 candidate should record at least the following metadata:

artifact:
  base_model: model-name@revision
  quant_scheme: fp8-w8a8
  quantizer: modelopt
  calibration_dataset: calib-prod-v3
  calibration_dataset_hash: sha256:...
  runtime: tensorrt-llm@version
  gpu: hopper
  kv_cache_dtype: same-as-dense-baseline

Candidate C: W4A16

If the core goal is reducing model weight VRAM, or if hardware FP8 support is suboptimal, build a W4A16 candidate. AWQ and GPTQ are both mature 4-bit weight-only approaches, but they differ algorithmically:

AWQ: Uses activation statistics to identify more important weight channels and applies scaling to reduce quantization loss on those weights;
GPTQ: Uses approximate second-order information to optimize quantization error layer by layer.

In production, there’s no need to argue over “which is absolutely better” upfront — test empirically against your model, hardware, and data.

3. Quality Gate: Don’t Just Look at Perplexity

A quantized model may show minimal change in overall language-modeling metrics while regressing noticeably on certain business request types. The release gate should therefore have at least three layers.

Layer 1: Core Capability Regression

Choose deterministic metrics based on model use case:

Task Type	Metric
Classification	Accuracy / F1
Extraction	Exact Match / F1
Math	Answer accuracy
Code	Executable test pass rate
Multiple Choice	Accuracy
Domain Tasks	Business-rule-decidable pass rate

If you can only use open-ended generation samples, at minimum keep a human-verified Golden Set and run deterministic checks on format, key facts, and mandatory items. Don’t treat “looks about the same” as a gate.

Layer 2: Key Slice Regression

Overall averages easily mask local problems. Build slices based on production traffic:

Chinese / English;
Short context / long context;
High-frequency business / low-frequency high-risk business;
Plain text / code / JSON;
Different system prompts or product lines.

The release condition isn’t “total score doesn’t drop” — it’s that no key slice exceeds the business-acceptable threshold. Thresholds shouldn’t be copied from papers; they should be defined jointly by product risk and cost objectives.

Layer 3: Behavioral Stability

The same batch of inputs should be compared across Dense and Quantized pipelines:

Is the refusal rate abnormally changed?
Does structured-format success rate drop?
Does generation length distribution drift?
Are EOS, repeated output, and abnormal truncation increasing?
Does accuracy on key terms and numbers change?

This layer is especially good at catching “benchmark scores unchanged, but production experience degraded” issues.

4. Performance Gate: Only Compare Under Identical Conditions

Performance experiments must keep the input set and serving parameters consistent. At minimum, record:

GPU VRAM usage;
Output tokens per second;
TTFT P50/P95/P99;
TPOT P50/P95/P99;
Maximum stable concurrency;
Throughput across input-length buckets;
Tail latency at different concurrency levels.

Use a fixed request-replay set, and version the request-length distribution, output-length distribution, and concurrency curve.

The most common mistake is running Dense with default parameters while enabling a different batch or kernel config for the quantized run to boost scores — making it impossible to tell whether gains came from quantization or serving-parameter changes.

5. Treat Quantized Artifacts as Traceable Software Artifacts

A production quantized model should be able to answer five questions:

Which Dense model version was it generated from?
Which quantization algorithm and recipe were used?
Which version of the calibration set was used?
Which quantizer/runtime version generated and runs it?
Which version of the eval report proves it meets release criteria?

You can design the artifact ID like this:

<model>@<revision>__<quant-scheme>__<calib-version>__<runtime-version>

For example:

model-x@r42__w4a16-awq__calib-v3__vllm-0.xx

The naming format isn’t the point — the point is that you can’t just keep a single quantized-model directory. If you can’t rebuild the same quantized checkpoint, you can’t reliably do regression testing or rollback.

6. Canary Rollout: Quantized Is a New Model, Not a New Config

Even if all offline gates pass, a quantized artifact should still be treated as a model-version release. The recommended flow is:

Dense remains the stable version;
Quantized first enters Shadow or internal traffic;
Then moves to a small Canary percentage;
Compare quality proxy metrics, error rates, TTFT/TPOT, throughput, and VRAM;
Gradually expand once targets are met;
Keep the Dense artifact and fast rollback capability until the full observation window ends.

Don’t delete the Dense version immediately after release. Quantization regression usually isn’t “the service is down” — it’s a slow quality degradation on a specific domain, language, or length slice, and these issues take observation time to surface.

7. When to Use Which Approach

FP8 is a better first choice when:

The GPU has mature hardware acceleration for FP8;
The goal is simultaneously reducing VRAM and increasing compute throughput;
The serving engine has validated the corresponding FP8 kernels;
You can accept the calibration, benchmarking, and quality-regression workflow.

W4A16 is a better first choice when:

Model weight VRAM is the primary constraint;
You want to reduce model size while keeping activations at high precision;
Deployment hardware has more mature support for AWQ/GPTQ/Marlin paths;
Request concurrency is low, but per-instance model loading cost is high.

Don’t rush into quantization when:

The current bottleneck is network, queueing, or business post-processing;
The model is already small enough and GPU utilization is chronically low;
You don’t have a stable business eval set;
You can’t keep a Dense rollback version;
The runtime only “loads” the target format without mature kernel support.

8. Common Misconceptions

Misconception 1: Download someone else’s pre-quantized checkpoint and ship it. Pre-quantized checkpoints are convenient, but you still need to verify the base revision, quantization recipe, hardware compatibility, and your own business quality. Being downloadable from Hugging Face doesn’t mean it fits your traffic.

Misconception 2: Bigger calibration sets are always better. The primary goal of a calibration set is to represent the real activation distribution. LLM Compressor’s docs suggest starting with a relatively small sample set and increasing data based on quality recovery. Blindly scaling up a dataset with the wrong distribution just calibrates the wrong target more expensively.

Misconception 3: Run one standard benchmark and call it done. Quantization error can concentrate in long context, code, numbers, low-resource languages, or a specific domain slice. You need both general benchmarks and a business Golden Set.

Misconception 4: Enable KV Cache quantization at the same time. This breaks attribution. In weight-quantization experiments, keep KV Cache config unchanged. If you later want to optimize KV Cache, set up a separate second experimental matrix.

Misconception 5: Only compare model file sizes. What actually matters is the stable business throughput per GPU and the cost under quality constraints. A smaller checkpoint with inefficient kernels doesn’t necessarily translate to production gains.

9. Rollout Checklist

10. FAQ

Should I choose between FP8 and W4A16? Don’t decide based on format alone. Fix the model version, GPU, serving engine, request set, and KV Cache config, then build separate FP8 and W4A16 candidates. FP8 emphasizes low-precision compute paths on modern GPUs; W4A16 leans toward weight compression. The final choice should be driven by both business quality and actual performance.

Can I directly use WikiText, C4, or UltraChat for calibration? They can serve as a starting point for general models, and the LLM Compressor docs give examples using these public datasets. But if your model serves a specific domain like code, Chinese finance, insurance, or law, prioritize adding or switching to data close to production inputs, covering real chat templates and length distributions.

Why doesn’t this article cover KV Cache quantization? Because it’s an independent variable. Weights, activations, and KV Cache differ in numerical error, VRAM gains, and kernel paths. Modifying multiple low-precision components in a single experiment makes quality and performance changes impossible to attribute. In production engineering, the safer approach is to quantize and gate one component at a time.

References

vLLM — Quantization: https://docs.vllm.ai/en/latest/features/quantization/
vLLM — INT4 W4A16: https://docs.vllm.ai/en/stable/features/quantization/int4/
LLM Compressor — Choosing your dataset: https://docs.vllm.ai/projects/llm-compressor/en/latest/steps/choosing-dataset/
LLM Compressor — Choosing the right compression algorithm: https://docs.vllm.ai/projects/llm-compressor/en/latest/steps/choosing-algo/
TensorRT-LLM — Quantization: https://nvidia.github.io/TensorRT-LLM/1.3.0rc12/features/quantization.html
TensorRT-LLM — FP8 Quantization: https://nvidia.github.io/TensorRT-LLM/performance/performance-tuning-guide/fp8-quantization.html
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: https://arxiv.org/abs/2306.00978