Does KV cache quantization always improve throughput?

Not necessarily. It primarily reduces memory footprint and memory bandwidth pressure. Long-context, decode-heavy, high-concurrency scenarios benefit most; short-context or prefill-dominated scenarios may see little gain or even slowdown due to quantization/dequantization overhead.

If I've already applied AWQ, GPTQ, or FP8 weight quantization, do I still need KV cache quantization?

They optimize different things. Weight quantization reduces model parameter memory; KV cache quantization reduces per-request cache memory during generation. Under long contexts and large batches, KV cache can become the new memory bottleneck.

What's the most important validation before going live?

You must validate long-context quality, TTFT, TPOT/ITL, throughput, peak memory, and failure cases with real business distributions. Prepare to fall back to BF16/FP16 KV cache per model, route, or tenant dimension.

How should I choose between FP8 and INT8?

Prioritize framework and hardware support. FP8 typically has more direct support on modern GPUs and attention kernels. INT8 can also reduce cache footprint; specific precision impact and kernel performance should be measured per framework.

KV Cache Quantization in Production: Reducing Long-Context Inference Memory with FP8/INT8

Background: The Bottleneck in Long-Context Inference Is Shifting from Weights to KV Cache

When many teams first optimize LLM inference, they focus on model weights—converting FP16 to INT8 or INT4, applying AWQ, GPTQ, or FP8, trying to fit larger models into GPUs or reduce per-forward-pass cost. This step is important, but as you move into long-context, multi-turn conversations, and high-concurrency serving, another bottleneck emerges faster: KV Cache.

Autoregressive models generate each token by attending to the Key/Value representations of all previous tokens. Without caching, you’d repeatedly recompute the history; with KV Cache, the model reuses historical Key/Values and only does incremental computation for new tokens. The problem is that the cache grows linearly with context length, batch size, number of layers, KV heads, and head dimension.

In short-prompt, short-output scenarios, KV Cache is negligible. But at 32K, 128K, or longer contexts—or when serving many concurrent requests on a single GPU—KV Cache directly consumes memory, limiting batch size, concurrency, and maximum context length. At this point, continuing to compress only model weights often fails to solve throughput problems.

KV Cache quantization has a straightforward goal: reduce the Key/Value cache from BF16/FP16 to FP8, INT8, or even lower bit-widths, shrinking cache size and attention read bandwidth, thereby improving long-context serving capability while maintaining acceptable quality.

Core Principles: You’re Quantizing the Model’s Historical Memory During Generation, Not the Weights

During Transformer decoding, each attention layer produces Keys and Values for historical tokens. The KV Cache size for a single request can be simplified as:

KV Cache bytes ≈ layers × sequence_length × kv_heads × head_dim × 2(K,V) × bytes_per_element

Here, bytes_per_element is the optimization target. BF16/FP16 is typically 2 bytes; FP8/INT8 is typically 1 byte. From a storage perspective alone, FP8/INT8 can roughly halve the KV Cache size. Real-world gains depend on kernels, scale storage, memory access patterns, quantization granularity, model architecture, and workload.

Why KV Cache Quantization Is Different from Weight Quantization

Dimension	Weight Quantization	KV Cache Quantization
Object	Model parameters (static)	Per-request runtime cache (dynamic)
Timing	Model loading / engine build	Online quantization during inference
Primary benefit	Reduces model memory, lowers forward pass cost	Reduces cache memory, lowers attention memory bandwidth
Typical methods	AWQ, GPTQ, FP8, INT4	FP8/INT8 KV Cache
Relationship	Can be combined, not interchangeable	Can be combined, not interchangeable

For example, even with INT4 weight quantization, long-context requests still generate large KV caches. Under high concurrency, memory may be dominated by the cache, not the weights. At that point, squeezing weights further has diminishing returns; compressing the KV Cache directly impacts maximum concurrency and context capacity.

FP8, INT8, 2-bit: Start with Production Viability, Not Just Compression Ratio

Papers often showcase 4-bit, 3-bit, or 2-bit KV Cache schemes—e.g., KIVI proposes asymmetric low-bit quantization with different granularity for Keys and Values; KVQuant, TurboQuant, RotateKV, and others explore even lower-bit cache compression. But engineering deployment should first distinguish two goals:

Stable production path: Prioritize FP8/INT8 KV Cache already supported by inference frameworks, paired with mature attention kernels.
Research or specialized optimization path: Sub-8-bit schemes offer higher compression but require careful evaluation of model, kernel, hardware, calibration, quality metrics, and framework support.

In production, don’t just ask “how many bits can we compress to?” Instead, ask:

Is it natively supported by our framework?
Are there high-performance kernels for quantized attention?
Is long-context accuracy stable?
Does it support per-layer skipping, per-model canary, per-route fallback?
Can we explain the impact on TTFT, TPOT, throughput, and quality?

Engineering Implementation: Start with Controllable Paths in vLLM/TensorRT-LLM

Step 1: Determine If You Really Need KV Cache Quantization

Use online or stress-test data to answer four questions:

What are the average and P95/P99 input lengths?
Are outputs mostly short answers, or long-chain reasoning / long report generation?
Is the GPU memory bottleneck from weights, activations, KV Cache, or batch scheduling fragmentation?
Is the current bottleneck TTFT, TPOT/ITL, throughput, OOM, or cost?

If most requests are short-prompt, short-output, KV Cache quantization may not be the top priority. You might need to optimize batching, prefix caching, continuous batching, speculative decoding, or weight quantization first.

Consider KV Cache quantization a priority when you see:

OOM errors as long-context requests increase
Inability to increase batch size
TPOT/ITL degrading noticeably with context length during decode
Persistent memory pressure even with prefix cache hits
GPU compute utilization is decent, but memory bandwidth or cache reads are the main bottleneck

Step 2: vLLM—Start with a Minimal Reproducible Experiment

vLLM supports FP8 KV Cache via the kv_cache_dtype flag. Start with no calibration or automatic scales, but supplement with real dataset calibration and quality evaluation before going live.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8

For some hybrid attention models, sliding window layers have small KV caches, and quantization overhead may not be amortized. Verify if skipping these layers is better:

vllm serve <model> \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window

If the model shows consistent quality degradation with uncalibrated FP8, use target-domain samples for scale calibration. vLLM’s llm-compressor examples provide a path for generating FP8 KV Cache scales from calibration data, suitable for more robust pre-production validation.

Step 3: TensorRT-LLM—Evaluate with Hardware and Engine Capabilities

TensorRT-LLM offers various KV Cache management features, including paged KV cache, quantized KV cache, circular buffer, and KV cache reuse. On NVIDIA Hopper, Ada, Blackwell, and similar hardware, the benefit of KV Cache quantization depends on engine version, attention backend, model architecture, and workload.

Don’t treat “enable FP8 KV Cache” as an isolated toggle. Evaluate it alongside:

Paged KV cache: Reduces fragmentation, improves memory management efficiency
KV cache reuse: Reuses system prompts, template contexts, or shared prefixes
Eviction policy: Prevents high-value cache entries from being evicted too early
Routing: Directs cache-hit requests to more suitable workers
Quantized cache: Reduces cache size and read bandwidth

Step 4: Quantization Scales—Don’t Overlook Calibration Strategy

KV Cache quantization typically requires scales to map high-precision values to low-precision representations. Different frameworks may support different granularities: per-tensor, per-head, per-channel, per-token, etc.

A common mistake is thinking setting kv_cache_dtype=fp8 is enough. You should compare at least three groups:

Group	Configuration	Purpose
Baseline	BF16/FP16 KV Cache	Quality upper bound reference
Experiment 1	FP8 KV Cache, no calibration or default scales	Quick compression benefit validation
Experiment 2	FP8 KV Cache, calibrated with real business data	Pre-production quality safety net

If Experiment 2 is significantly better than Experiment 1, the model or business distribution is sensitive to scales, and you can’t rely on default configurations in production.

Evaluation Metrics: Don’t Just Look at Throughput; Also Look at Long-Context Quality

KV Cache quantization is often mis-evaluated. Looking only at average throughput on single-turn short prompts can completely miss issues. Cover at least the following metrics.

Performance Metrics

Metric	Description
TTFT	Time to first token. When long-context prefill is heavy, FP8 attention may introduce extra overhead; test separately
TPOT / ITL	Per-token latency during generation. Main benefit in long-context decode-heavy scenarios
Throughput	tokens/s, requests/s, effective concurrency
Peak memory	Total memory, KV Cache usage, fragmentation rate
OOM rate	Especially under combined long-context and high-concurrency conditions

Quality Metrics

Long-context retrieval: Needle-in-a-haystack, MRCR, RULER, or your own long-context QA datasets
Factual consistency: Citation-based QA, source matching in RAG scenarios
Reasoning stability: Math, code, long-chain reasoning
Output degradation: Repetition, truncation, off-topic responses, format drift
Bucket evaluation: Break down by context length (8K, 32K, 64K, 128K, 256K, etc.)

Online monitoring shouldn’t rely on averages alone. KV Cache quantization issues often appear in long-tail requests, extremely long contexts, specific model layers, or unusual business text.

Applicable Scenarios

Scenarios Where KV Cache Quantization Is More Beneficial

Long-context Q&A, long document summarization, codebase understanding
Multi-turn agents with growing historical context
High-concurrency online inference where batch size is memory-bound
Decode-heavy tasks like long answers, long reasoning, long report generation
GPU memory is tight, but you want to increase concurrency or context length

Scenarios Requiring Caution

Short-context, low-concurrency services where quantization overhead may outweigh benefits
Businesses extremely sensitive to fine-grained numerical values, long-chain reasoning, or high-precision text (legal, medical)
Non-standard attention backends, or frameworks with just-released support for the model
Large head_dim and prefill latency is a core SLA
Lack of long-context test sets to detect quality regression

Common Misconceptions

Misconception 1: Weight Quantization Eliminates the Need for KV Cache Quantization

Weights and KV Cache are two different memory regions. Weight quantization makes the model smaller but doesn’t automatically reduce the per-request KV Cache that grows during generation. In long-context scenarios, you typically need to evaluate both together.

Misconception 2: Stress-Testing Only with Short Prompts

Short prompts won’t reveal the linear growth of KV Cache or long-context attention errors. You must evaluate by context length buckets before going live.

Misconception 3: Only Looking at Throughput, Ignoring Quality

Quantization gains are easily amplified by tokens/s, but quality degradation may only appear under extremely long contexts, specific tasks, or particular model architectures. Don’t deploy throughput improvements without quality evaluation.

Misconception 4: Default Scales Are Always Sufficient

Default scales are simple and reproducible, but not all models are stable with them. If you see systematic quality degradation, introduce calibration data or disable KV Cache quantization for that model/route.

Misconception 5: Quantizing All Layers Is Always Optimal

Hybrid attention or sliding window layers have limited cache size; full-layer FP8 may not be optimal. Some frameworks support skipping specific layers—include this in your experiment matrix.

Pre-Deployment Checklist

Summary

KV Cache quantization is a critical step for long-context LLM inference—moving from “it runs” to “it runs well in production.” It’s not a silver bullet, but in the right scenarios—long context, high concurrency, decode-heavy—it can significantly reduce memory pressure and increase service capacity. The core principle: anchor on the BF16 baseline, measure with calibration data, evaluate with bucket analysis, deploy with canary, and always be ready to roll back.

Don’t chase 2-bit/4-bit paper schemes right away. First, get FP8/INT8 stable in your production framework, understand your workload characteristics, build a quality monitoring system, and then consider more aggressive compression strategies.