Why does my inference service get CUDA OOM when nvidia-smi shows free memory?

OOM depends not only on total free memory but also on whether the allocator can find a usable block for the current request. Dynamic batching, varying sequence lengths, and temporary workspaces can leave non-reusable fragments. Additionally, NCCL or direct CUDA allocations may not appear in the PyTorch Snapshot.

Can torch.cuda.empty_cache() fix memory fragmentation?

It only returns completely free cache segments to CUDA; it cannot free memory still held by tensors or guarantee fixing all fragmentation. Calling it on every request's hot path typically breaks cache reuse and increases latency.

Should I enable expandable_segments or cudaMallocAsync directly in production?

No, not blindly. Both require A/B testing based on real shape distributions, inactive split blocks, throughput, and tail latency. Some native allocator parameters are ignored under the cudaMallocAsync backend.

Is it safe to continue serving after an OOM?

It depends on whether the framework can fully clean up the failed request's state. A caught single-request OOM does not guarantee that all custom operators, communication states, or CUDA Graphs have been restored. On consecutive OOMs, mark the worker as not accepting new traffic, drain it, and restart.

LLM GPU Memory Fragmentation in Production: Taming OOMs with Memory Snapshots, Allocator Tuning, and Admission Control

Background: OOM Doesn’t Always Mean Memory Is “Full”

GPU memory in an LLM inference process typically hosts model weights, KV Cache, temporary activations and operator workspaces, CUDA Graph private memory pools, and communication buffers. When request batch sizes, input lengths, output lengths, or multimodal dimensions vary continuously, the allocator must repeatedly request and reuse blocks of different sizes.

The common failure mode here is not absolute total capacity exhaustion, but:

memory_reserved is high, but memory_allocated (actually used by tensors) is low;
Numerous scattered holes within cache segments that cannot satisfy new requests;
A single large contiguous allocation needed for Prefill or a temporary Workspace triggers OOM;
The PyTorch Snapshot shows remaining headroom, but NCCL, third-party kernels, or direct CUDA API calls occupy invisible memory.

Therefore, production management cannot just watch nvidia-smi, nor can it simplify OOM to “just lower gpu_memory_utilization a bit.” The correct sequence is: first classify, then locate, then tune the allocator and request shapes, and finally establish OOM admission control and recovery strategies.

Core Principles: First Distinguish Four Types of Memory Problems

1. True Capacity Shortage

The sum of weights, KV Cache, execution workspace, and safety margin is close to the GPU’s total capacity. In this case, no amount of defragmentation can accommodate new peak requests.

Typical signals: both allocated and reserved are consistently near their limits, and OOMs are strongly correlated with maximum concurrent tokens, longest context, or largest image size.

2. Lifecycle Leaks

Certain tensors, request contexts, hooks, or asynchronous futures are unexpectedly retained, causing active memory to grow monotonically with requests without falling back. This differs from fragmentation: leaks show a steady increase in active allocation, not increasingly scattered free blocks.

3. Allocator Fragmentation

PyTorch uses a caching allocator to avoid frequent device synchronization. Freed blocks are kept within the process for future reuse, so what nvidia-smi shows is usually closer to reserved than the current tensor allocated.

When dynamic batches switch between N, N-1, N+1, allocation sizes from different layers also change. The tail of old segments can leave many non-mergeable fragments, leading to a situation where “total free space seems sufficient, but no suitable block exists.”

4. Allocator-External Memory

The PyTorch Memory Snapshot only shows memory managed by the PyTorch allocator. Memory allocated directly by NCCL, certain custom CUDA extensions, driver contexts, or third-party inference libraries may not appear in the Snapshot.

If the total device usage is significantly higher than PyTorch’s reserved, the difference must be monitored separately as external GPU memory and not mistaken for a PyTorch leak.

Engineering Practice 1: Establish a Memory Observation Baseline

Before going live, collect at least the following metrics:

torch.cuda.memory_allocated() and its peak;
torch.cuda.memory_reserved() and its peak;
The count and byte size of inactive split blocks from memory_stats();
The difference between total GPU device usage and PyTorch reserved;
Request-side Batch, Prompt Token, Expected Output Token, and modality dimensions;
The request type, concurrent tokens, instance runtime, and most recent shape change at the time of OOM.

Define two derived metrics:

allocator_slack = reserved_bytes - allocated_bytes
external_bytes  = device_used_bytes - reserved_bytes

A high allocator_slack doesn’t necessarily mean fragmentation, but if it coincides with a large number of inactive split blocks and OOM occurs during a large allocation request, further analysis is warranted.

Engineering Practice 2: Use Memory Snapshots to Find the Allocation Timeline

PyTorch provides Memory Snapshots to record allocation, free, segment allocation, and OOM events along with their call stacks. This is suitable for pre-release load testing, fault replicas, or short diagnostic windows.

import torch

def run_probe() -> None:
    # Replace with inference replay using realistic length distributions and concurrency patterns
    ...

torch.cuda.memory._record_memory_history(max_entries=100_000)
try:
    run_probe()
except torch.cuda.OutOfMemoryError:
    torch.cuda.memory._dump_snapshot("/tmp/llm-oom-snapshot.pickle")
    raise
finally:
    torch.cuda.memory._record_memory_history(enabled=None)

When analyzing the Snapshot, focus on three things:

Whether there are many inactive blocks just before the OOM;
Whether new shapes are constantly creating new segments slightly larger than old blocks;
Whether there are active allocations that should have been freed with the request but persist.

Snapshots can be large; do not enable them indefinitely on all production replicas. A safer approach is to control them with a diagnostic switch, limiting the number of events, runtime, and output directory.

Engineering Practice 3: Govern Shapes First, Then Tune the Allocator

Allocator tuning cannot replace request governance. For inference services, the most effective first step is usually to reduce infinitely discrete shapes:

Constrain Batch Size to a limited set of values;
Bucket Prompt Length, Max Output Tokens, and multimodal resolutions;
Route very long requests separately to avoid mixing with normal requests;
Use concurrent tokens, not request count, for Admission Control;
Reserve additional headroom for models or kernels with large estimated peak workspaces.

Shape Buckets are not about having as few as possible. Too coarse buckets cause padding waste; too fine buckets reduce memory block reuse. Use real traffic replay to find the balance between padding cost, cache reuse, and tail latency.

Engineering Practice 4: Choose Allocator Parameters Based on Evidence

PyTorch currently supports a native caching allocator and a CUDA asynchronous allocator. Parameters should be tested individually, not all enabled at once.

Option A: Dynamic Batching is Significant, Evaluate `expandable_segments`

export PYTORCH_CUDA_ALLOC_CONF="backend:native,expandable_segments:True"

expandable_segments aims to let segments grow with demand, reducing the tail fragmentation left by dynamic batching across multiple network layers. It is still experimental and must be validated for compatibility with your GPU model, PyTorch version, CUDA Graphs, and third-party operators.

Option B: High Inactive Split Blocks, Try `max_split_size_mb` Last

export PYTORCH_CUDA_ALLOC_CONF="backend:native,max_split_size_mb:256"

This parameter limits the further splitting of large blocks, which may alleviate boundary OOMs, but the performance impact depends on the allocation pattern. The official documentation explicitly recommends it as a last resort, adjusted using memory_stats() and memory_summary().

Option C: Evaluate `cudaMallocAsync`

export PYTORCH_CUDA_ALLOC_CONF="backend:cudaMallocAsync"

The CUDA asynchronous allocator is based on the Stream Ordered Memory Allocator and requires a compatible CUDA version. After switching, some statistics and parameters specific to the native backend become meaningless, such as max_split_size_mb, roundup_power2_divisions, and garbage_collection_threshold.

A/B Validation Dimensions

Each option should be A/B tested individually. “No OOM” is not the only success criterion. Also compare:

Dimension	Description
P50/P95/P99 TTFT and TPOT	Time to First Token and Time Per Output Token
Throughput and GPU Utilization	Whether overall service capacity drops
reserved/allocated difference	Change in caching efficiency
Fragmentation trend under prolonged mixed traffic	Whether it worsens over time
CUDA Graph capture and replay	Whether it adds extra private pools
Worker restart frequency and single request failure rate	Stability metrics

Engineering Practice 5: Reserve Invisible and Burst Memory for vLLM

vLLM’s --gpu-memory-utilization is a per-instance budget, not a global coordinator for the entire GPU. The current development documentation default is 0.92, but production environments should re-determine this value based on the specific version, model, parallelism, and co-located processes.

For precise control over KV Cache, use --kv-cache-memory-bytes, which overrides the automatic inference based on gpu_memory_utilization. Regardless of the method, do not allocate all remaining memory to the KV Cache. Reserve for:

Prefill and sampling temporary workspaces;
CUDA Graph private pools;
NCCL or custom operator memory;
Burst shapes and diagnostic tool overhead;
Driver, context, and changes from other processes on the same GPU.

Example startup parameters:

vllm serve /models/your-model \
  --gpu-memory-utilization 0.86 \
  --max-model-len 32768 \
  --max-num-batched-tokens 16384

These values are only configuration examples, not universal recommendations. The correct values must come from capacity tests on your target hardware and real traffic replay.

OOM Admission Control: Let a Single Anomalous Request Fail, Not the Worker

Production systems should estimate the peak cost of a request before it enters the scheduler. A simple model can be:

request_peak ≈ kv_growth + prefill_workspace + multimodal_encoder_workspace + safety_margin
admit only if request_peak <= reclaimable_headroom

The estimate doesn’t need to be perfectly accurate initially, but it must be conservative and continuously calibrated with OOM samples. At a minimum, implement:

Preemptively reject requests exceeding token, image count, or resolution budgets;
On OOM, fail only the current request and log the complete request shape;
On consecutive OOMs or worsening fragmentation metrics, stop accepting traffic and drain the worker;
Worker restart as a final recovery mechanism, not a daily cleanup strategy;
Do not call empty_cache() after every request.

Newer PyTorch allocator configurations also offer OOM-throwing options for inference services, allowing the framework to catch a single-request OOM and continue serving; whether to use this depends on validation with your current PyTorch and serving framework versions.

Applicable Scenarios

This approach is suitable for:

Online inference with dynamic batching and mixed long/short requests;
Multimodal services with mixed text, image, and video inputs;
Processes where CUDA Graphs, Tensor Parallelism, or custom kernels share GPU memory;
Multiple models or serving instances sharing the same GPU;
Services that only start experiencing OOM after several hours of runtime.

If the model weights themselves already exceed physical GPU memory, prioritize quantization, parallelism, CPU offloading, or model selection; allocator tuning cannot solve fundamental capacity shortages.

Common Misconceptions

Misconception	Correct Practice
Only watch `nvidia-smi`	`nvidia-smi` cannot distinguish memory in use by tensors, PyTorch cache, NCCL allocations, or other processes. Must combine `allocated`, `reserved`, Snapshots, and device-level differences.
Treat all `reserved - allocated` as fragmentation	This includes normally reusable cache. Only when block distribution, inactive splits, and failed allocations point to non-reusability can it be diagnosed as fragmentation.
Call `empty_cache()` on every OOM	`empty_cache()` can only return completely free segments; it cannot free active tensors. Frequent calls also increase reallocation and synchronization overhead.
Set `gpu_memory_utilization` close to 1	There are still execution workspaces, communication libraries, Graph private pools, and external allocations outside the inference engine. Over-compressing headroom turns occasional spikes into systematic OOMs.
Enable all allocator parameters at once	Parameters have backend constraints, and their effects depend on shape distribution. Without single-variable A/B testing, you cannot determine the source of gains or safely roll back.

Go-Live Checklist

References

PyTorch — CUDA semantics / Memory management
https://docs.pytorch.org/docs/2.13/notes/cuda.html
PyTorch — Understanding CUDA Memory Usage
https://docs.pytorch.org/docs/2.13/torch_cuda_memory.html
PyTorch Blog — Understanding GPU Memory: Visualizing All Allocations over Time
https://pytorch.org/blog/understanding-gpu-memory-1/
NVIDIA CUDA Runtime API — Memory Pools
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY__POOLS.html
vLLM — Engine Arguments
https://docs.vllm.ai/en/latest/configuration/engine_args/