LLM Serving CUDA Graph in Production: Taming Kernel Launch Jitter with Shape Buckets, Piecewise Capture, and Warmup Plans

When optimizing LLM serving performance, engineers typically look first at GPU utilization, attention kernels, KV cache, and batch size. But on short decode steps, small batches, or smaller models, another common bottleneck is CPU-to-GPU kernel launch overhead.

An autoregressive decode step isn’t a single GPU call—it executes a long chain of operators. Each operator requires parameter preparation and kernel submission across the Python/C++ runtime, CUDA driver, and GPU. When individual kernels are fast, the submission overhead itself becomes a significant fixed cost.

This article draws on vLLM and TensorRT-LLM to explain shape buckets, piecewise capture, warmup plans, memory trade-offs, and production rollback gates—helping you eliminate the host-side bottleneck where “the GPU isn’t saturated, but latency won’t drop.”

Core Principle: CUDA Graph Optimizes Launch Overhead, Not Compute

1. What Graph Replay Eliminates

The normal execution path looks roughly like this:

CPU prepare op A -> launch kernel A
CPU prepare op B -> launch kernel B
CPU prepare op C -> launch kernel C
...

After CUDA Graph capture, it becomes:

copy/update static inputs
        ↓
cudaGraphLaunch(graph)
        ↓
GPU replays A -> B -> C -> ...

CUDA Graph is therefore best suited for host-bound inference phases. If a phase is already fully compute-bound—where GPU kernel duration far exceeds launch overhead—CUDA Graph may still help, but the gains are typically less dramatic than with short kernels and low-batch decode.

2. Why Dynamic Shapes Are the Core Tension

CUDA Graph depends on a stable execution graph and memory layout. A graph captured for batch=8 cannot simply be treated as a dynamic program for batch=37.

Serving frameworks typically use two strategies:

1. Capture multiple discrete shapes/batch sizes.
2. At runtime, pad the actual shape up to the nearest captured bucket.

This is the essence of shape buckets. Suppose your production decode batch distribution clusters around 1, 2, 4, 8, 12, 18, and 30. You might design:

Runtime Batch	Handling
12	pad / replay 16
18	pad / replay 32
40	eager fallback / other path

Too few buckets waste compute on padding; too many buckets inflate capture count, static buffers, and memory usage. The bucket grid is fundamentally a cost function trading off latency, extra compute, capture time, and memory.

Full CUDA Graph vs. Piecewise CUDA Graph

Full Capture: Highest Reward, Strongest Constraints

If the entire model forward pass satisfies CUDA Graph requirements, you can use Full CUDA Graph. vLLM distinguishes between Full, Piecewise, Full Decode Only, and Full + Piecewise modes, with Full Decode being especially relevant for pure decode workloads.

However, attention is one of the hardest regions to capture in an LLM. Its shapes, KV state, backend capabilities, and control paths can all affect graph safety. The claim “full graph is always faster” cannot be made independently of the model, attention backend, and workload.

Piecewise Capture: Keep Hard-to-Capture Operators on the Eager Path

Both vLLM and TensorRT-LLM support the piecewise CUDA Graph approach:

CUDA Graph Segment A
        ↓
Attention / dynamic op (eager)
        ↓
CUDA Graph Segment B
        ↓
Attention / dynamic op (eager)
        ↓
CUDA Graph Segment C

This sacrifices some full-graph benefit for better dynamic compatibility. TensorRT-LLM explicitly notes that piecewise CUDA Graph keeps regions that are hard to capture—especially attention—on the eager path while capturing the rest. It configures capture_num_tokens by token count, padding the actual token count up to the next captured value at runtime.

The real tuning target in production isn’t a boolean switch; it’s the graph partition + capture grid.

Shape Buckets Should Be Derived from Production Traffic, Not Copied from Defaults

Many frameworks ship default capture sizes. Defaults are fine for “out-of-the-box” use, but they aren’t necessarily optimal for your workload. A more reliable approach is to collect a workload histogram over 24 hours or 7 days:

metric: active_decode_sequences
metric: num_scheduled_tokens
metric: prefill_tokens_per_iteration
metric: mixed_batch_tokens

Then compute coverage for each candidate bucket. For example:

Range	Share
1-4 tokens/sequences	31%
5-8	27%
9-16	21%
17-32	13%
33-64	6%
>64	2%

Such traffic should have denser buckets at smaller shapes, not a uniform spread.

A Practical Bucket Selection Heuristic

A reasonable candidate bucket set might be:

1, 2, 4, 8, 12, 16, 24, 32, 48, 64, 96, 128

Then compare against a real trace using:

• Graph coverage
• Padding ratio
• Graph memory
• p95/p99 TTFT
• p95/p99 TPOT
• Max concurrency

Don’t compare only tokens/s. vLLM’s current configuration logic also uses discrete capture sizes, selecting the appropriate CUDA Graph for batches it can cover at runtime; requests exceeding the maximum capture size don’t use the graph path. Framework defaults serve as a baseline, but production configuration should be calibrated to your real distribution.

Warmup Plan: Don’t Leave Compile/Capture Latency to Your First Real User

The most easily overlooked CUDA Graph issue in test environments is first-request cost. When a new shape is encountered for the first time, the system may trigger:

1. Torch/Inductor compilation
2. Lazy CUDA initialization
3. Attention backend initialization
4. Workspace allocation
5. CUDA Graph warmup
6. Capture and instantiate
7. Static buffer setup

PyTorch’s official CUDA Graph documentation explicitly recommends warmup before capture to avoid baking lazy initialization into the graph or causing capture failures. Production deployment shouldn’t rely on a single “health check” request—you need a shape-aware warmup plan:

warmup_plan:
  decode_buckets: [1, 2, 4, 8, 16, 32]
  mixed_token_buckets: [64, 128, 256, 512]
  repeat_per_bucket: 3
  block_ready_until_complete: true

Ready Doesn’t Mean Process Started

Kubernetes and service registration layers should distinguish:

Process Started
   ↓
Model Loaded
   ↓
Compile Complete
   ↓
Required Graph Buckets Captured
   ↓
READY

Otherwise, during rolling deployments, new instances join the load balancer and treat real user traffic as warmup requests.

More Captures Don’t Mean Less Memory

CUDA Graph is often described as “reducing CPU overhead,” but production configuration must also account for graph memory footprint. Capturing more shapes typically means more graph executables, static input/output buffers, and associated runtime state. TensorRT-LLM also explicitly notes that a wider capture token count increases GPU memory and can reduce achievable concurrency.

So CUDA Graph and KV cache must be considered within the same memory budget:

GPU Memory
├── Model Weights
├── KV Cache
├── Runtime Workspace
├── CUDA Graph / Static Buffers
└── Safety Margin

If pushing graph coverage from 97% to 99.8% reduces concurrent sequences by 15%, that optimization may not be worthwhile. A better metric: effective concurrency and effective tokens/s per GB of memory while meeting tail SLOs.

vLLM: Control the Capture Grid First, Then Evaluate Full/Piecewise

Here’s a simplified illustration—check your deployment version’s docs for exact fields:

vllm serve /models/your-model \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16,32,64]}'

Don’t chase the “largest capture grid” from the start. A safer sequence:

1. Run baseline in eager/default mode.
2. Collect real batch/token shapes.
3. Cover 90–95% of high-frequency shapes first.
4. Measure graph hits and padding.
5. Observe memory and concurrency changes.
6. Decide whether to add larger capture sizes.

vLLM’s current docs also describe Full, Piecewise, Full Decode Only, and Full + Piecewise CUDA Graph modes. Support evolves quickly across versions, so the runtime version must be recorded as part of the benchmark artifact.

TensorRT-LLM: Token Count Buckets Matter More Than a Single max_batch_size

TensorRT-LLM’s piecewise CUDA Graph configuration looks like:

torch_compile_config:
  capture_num_tokens: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
  enable_userbuffers: false
  enable_piecewise_cuda_graph: true

Official docs highlight:

• Piecewise capture token buckets should be tuned for your hardware, model, and parallelism strategy.
• More buckets reduce padding but increase graph memory.
• In the context phase, larger token counts already reduce host overhead’s share, so adding more capture points has diminishing returns.

A sensible bucket density rule: dense at small shapes, sparser as shapes grow.

Production Observability: Add at Least Six CUDA Graph Metrics

If you only watch TTFT and GPU utilization, it’s hard to explain why latency occasionally spikes after enabling graphs. Add these:

1. Graph Hit Ratio: cuda_graph_hit_requests / eligible_requests, ideally bucketed.
2. Eager Fallback Ratio: share of requests falling back to eager due to out-of-range shapes, backend incompatibility, or graph invalidation.
3. Padding Overhead: padding_ratio = padded_tokens / actual_tokens.
4. Capture/Compile Latency: distinguish model_load_ms / compile_ms / warmup_ms / capture_ms / ready_ms—critical for rolling scale-outs and failure recovery.
5. Graph Memory Footprint: record GPU used memory, KV cache capacity, and max concurrent sequences before and after enabling.
6. Tail Latency: at minimum compare TTFT p50/p95/p99, TPOT/ITL p50/p95/p99, and E2E p95/p99.

CUDA Graph’s value typically shows up more in small-batch decode and tail latency than in average throughput.

Recommended Rollout Process

Phase 1: Establish an eager baseline. Fix model revision, runtime version, CUDA/driver, GPU model, tensor parallel/expert parallel, and workload trace. Otherwise benchmarks aren’t comparable.

Phase 2: Enable only default CUDA Graph. Confirm the real benefit and memory cost of the framework’s default configuration.

Phase 3: Tune the capture grid with a production trace. For each bucket set, record coverage / padding_ratio / graph_memory / TTFT p99 / TPOT p99 / max_concurrency.

Phase 4: Add a warmup gate. New pods/workers must complete capture of critical buckets before becoming ready.

Phase 5: Canary. Don’t use average throughput as the sole rollout gate. At minimum, set:

• TTFT p99 regression <= threshold
• TPOT p99 regression <= threshold
• OOM = 0
• Graph fallback ratio <= threshold
• Max concurrency regression <= threshold

Phase 6: Keep an eager/previous-runtime rollback path. The compatibility matrix across CUDA Graph, Torch Compile, and attention backends shifts with versions. Your release system must be able to roll back to the previous runtime artifact.

When This Applies

This approach is especially relevant for:

• Online chat with high decode share
• Small models or low-batch, low-latency services
• Scenarios where the GPU isn’t continuously saturated but CPU launch overhead dominates step gaps
• Production systems with a stable request shape distribution
• Serving platforms that have completed KV cache, batching, and kernel optimizations and want to push tail latency lower

If your service is dominated by very large prefills where individual kernels already saturate the GPU for long stretches, CUDA Graph may not be the highest-priority optimization.

Common Misconceptions

Myth 1: More capture sizes are always better. More graphs improve coverage but increase capture cost and memory. The goal should be the minimal graph set covering most production traffic.

Myth 2: Full CUDA Graph is always better than Piecewise. Full capture is only worthwhile when the attention backend, dynamic shapes, and control flow all meet the requirements. Production systems value stability and compatibility more.

Myth 3: One warmup request after startup is enough. If you have multiple shape buckets, warming up only one shape won’t prevent capture spikes when other buckets are hit for the first time.

Myth 4: Only watch tokens/s. If tokens/s rises 5% but TTFT p99, OOM, or max concurrency degrades, that may not be a production win.

Myth 5: Ignore runtime versions. vLLM, TensorRT-LLM, and PyTorch continue to evolve their CUDA Graph implementations. Capture modes, compatible attention backends, default buckets, and memory policies can all change. Runtime version should be versioned just like model revision.

Pre-Launch Checklist

Before going live, confirm at least:

• A real workload shape histogram exists
• Capture buckets come from traffic distribution, not copied examples
• Graph hit ratio and eager fallback are measured
• Padding overhead is measured
• GPU memory and KV cache capacity are recorded before and after capture
• Warmup covers all critical buckets
• Readiness only allows traffic after warmup/capture completes
• TTFT/TPOT/ITL p95 and p99 are compared
• An eager or previous-runtime rollback path exists
• Runtime, driver, CUDA, model version, and configuration are all part of the release artifact

References

1. vLLM Compilation Configuration
2. vLLM Compilation API
3. vLLM Runtime / CUDA Graph Capture Size Configuration
4. TensorRT-LLM — Torch Compile & Piecewise CUDA Graph
5. PyTorch — CUDA Semantics / CUDA Graphs
6. Torch-TensorRT — CUDAGraphs