LLM Cold Start Optimization in Production: Reducing Model Load Time with Local NVMe Cache, Weight Streaming, and Warm Pools

Many teams optimizing LLM serving focus on HPA, KEDA, queue length, GPU utilization, and replica counts. These mechanisms answer “when to scale and by how much,” but in the large-model world, a newly scheduled Pod on a GPU node still faces a long startup chain before it’s truly Ready:

Pod Scheduling -> Container Image Ready -> Model Weights Fetch -> Weights Read / Deserialize -> CPU to GPU Transfer -> Runtime Initialization -> CUDA Graph / Kernel Warmup -> Readiness

So the real metric isn’t a single Pod’s startup time — it’s the total cold start duration:

cold_start_total = scheduling_time + image_prepare_time + weight_fetch_time + weight_load_time + runtime_init_time + warmup_time

For models that are tens or even hundreds of GB, weight_fetch_time and weight_load_time are often the dominant terms. If every scale-out re-downloads the model from object storage or Hugging Face Hub, then even if Kubernetes schedules the Pod in seconds, the business still sees minute-level latency before the service is usable.

This is the boundary between cold start governance and autoscaling governance: Autoscaling decides “whether we need a new replica”; Cold Start Engineering decides “how quickly that replica can actually serve inference.”

Core Principle: Bring Weights Closer to the GPU and Reduce Serial Waits in the Loading Path

1. Use Local NVMe to Turn “Remote Download” into “Node-Local Read”

KServe’s Local Model Cache directly addresses this problem: it pre-caches model weights to the local disk of GPU nodes, typically NVMe. When an InferenceService starts, if the target node already has the model, it reads from the local cache instead of re-downloading from a remote repository.

KServe’s current model cache design includes resources like LocalModelCache, LocalModelNamespaceCache, LocalModelNodeGroup, and LocalModelNode. The namespace-scoped cache also supports multi-tenant isolation, preventing unbounded cache sharing across all namespaces.

A simplified configuration looks like this:

apiVersion: serving.kserve.io/v1alpha1
kind: LocalModelCache
metadata:
  name: qwen-production-model
spec:
  sourceModelUri: "hf://your-org/your-model"
  modelSize: 40Gi
  nodeGroups:
    - gpu-workers

When actually implementing this, the key isn’t just “whether a cache exists” — you need to define clearly:

• Which models must be pre-warmed to all GPU nodes;
• Which models are cached only on a subset of nodes;
• What gets evicted first when node disk space runs low;
• When old caches become invalid after a new model version is released;
• Whether to fall back to the source when model file validation fails.

Without these rules, a local cache can quickly turn from a cold-start optimization into a disk capacity incident.

2. Use Weight Streaming to Shorten the “Read File → CPU → GPU” Serial Path

Even if weights are already in object storage or a high-speed file system, traditional loading often goes through multiple stages: full download or mmap first, then CPU reads, then gradual transfer to GPU.

vLLM now supports the Run:ai Model Streamer. The core idea is to read tensors concurrently and stream weights directly into the GPU. It also supports loading directly from object stores like S3, GCS, and Azure Blob, with tunable read concurrency, CPU buffer size, and distributed streaming.

For example:

vllm serve s3://llm-models/qwen-prod \
  --load-format runai_streamer \
  --model-loader-extra-config '{"concurrency":16}'

For multi-GPU models, you can also evaluate distributed streaming or pre-sharded checkpoints so each rank loads closer to “only the weights it actually needs.”

One common misconception to avoid: Weight Streaming isn’t always faster in every environment. The actual benefit depends on object storage throughput, network bandwidth, CPU memory bandwidth, GPU H2D bandwidth, checkpoint format, and concurrency parameters. Before production, benchmark with the real model, real nodes, and real storage — don’t blindly copy a fixed concurrency value.

3. Warm Pools Shouldn’t Be Just “Always-On GPUs”

Full Scale-to-Zero is the cheapest, but it leaves all cold-start stages to the first request. Keeping every model resident on GPU creates significant idle cost. A more practical approach is to classify models into multi-level temperature states:

Tier	State	Cost Profile
Tier 0	Active GPU Replica	Highest cost, fastest startup
Tier 1	CPU-resident / Sleeping Replica	Medium cost, faster wake-up
Tier 2	Node-local NVMe Model Cache	Low cost, requires reload
Tier 3	Remote Object Storage	Lowest cost, slowest fallback

Different models enter different tiers based on SLO, call frequency, and startup cost. For example, high-frequency core models maintain a few GPU warm replicas; medium-frequency models can keep containers and CPU-side weights; low-frequency long-tail models keep only node-level NVMe caches; and extremely low-frequency models fall back entirely to remote object storage.

vLLM’s Sleep Mode provides a valuable implementation approach for Tier 1. Level 1 offloads model weights to CPU and discards the KV cache, freeing significant GPU memory. The model can then be woken up without re-fetching the same weights from remote.

However, note that vLLM’s documentation explicitly states that the online Sleep/Wake control endpoints require development mode. In production, don’t expose these endpoints directly to the public internet or tenant side. Instead, they should be invoked by an internal lifecycle controller with proper permissions, timeouts, and state machines.

4. Consider “Where the Model Is” in Scheduling, Not Just “Which GPU Is Free”

The most overlooked layer in cold-start optimization is placement. Suppose the cluster has two idle GPU nodes:

• Node A already has the target model cached on local NVMe;
• Node B has no trace of the model;
• Both machines have identical GPU models, memory, and current utilization.

A traditional scheduler might randomly pick B. From a GPU resource perspective, that’s fine — but from a model startup time perspective, it’s clearly suboptimal.

A key design in ServerlessLLM is startup-time-aware scheduling: it selects servers with shorter expected startup times based on the locality state of checkpoints on different servers, rather than just looking at abstract compute resources.

In a Kubernetes environment, this idea can be reduced to a simple rule:

Score(node, model) = GPU_fit_score + model_locality_score - remote_fetch_penalty - cache_pressure_penalty

Model locality should be a scheduling signal, not a fact discovered passively after startup.

Engineering Implementation: Build an Observable Cold Start State Machine

Instead of just recording a single Pod Ready time, break each model replica’s startup into observable states:

PENDING_GPU -> IMAGE_READY -> CACHE_LOOKUP -> FETCHING_WEIGHTS -> LOADING_WEIGHTS -> RUNTIME_INIT -> WARMING_UP -> READY

At minimum, collect the following metrics:

• cold_start_total_seconds: total time from scale decision to Ready;
• model_fetch_seconds: time to fetch weights from remote;
• model_load_seconds: time to read and load weights;
• local_model_cache_hit_rate: node-local model cache hit rate;
• remote_weight_bytes: bytes read from remote per startup;
• warm_pool_hit_rate: whether a scale-out request hits the warm pool;
• sleep_wakeup_seconds: time for a sleeping replica to return to Ready;
• cold_start_failure_rate: percentage of failures from download, validation, OOM, and initialization.

If you only monitor pod_startup_latency, you only know “it’s slow.” Once you break it down, you can tell whether the bottleneck is the Scheduler, Registry, S3, NVMe, CPU reads, H2D transfer, or runtime warmup.

A Recommended Production Strategy

You can adopt a three-tier strategy based on model heat and SLO.

Tier	Strategy Highlights
High-frequency core models	Keep 1–N Active Warm Replicas; pre-cache models to NVMe on all target GPU nodes; prioritize scheduling to cache-hit nodes; optimize p95/p99 cold start time rather than chasing Scale-to-Zero.
Medium-frequency models	Allow GPU scale-down; prefer keeping CPU-resident / Sleeping state; keep full weights on node NVMe; re-acquire GPU quickly when needed.
Long-tail models	No GPU warm replicas; cache only the most-used versions on a subset of NVMe nodes; allow fallback to object storage for extremely low-frequency models; expose a more relaxed cold-start SLO to callers.

Essentially, this strategy is a tiered trade-off between GPU cost, CPU memory, NVMe capacity, network traffic, and startup latency — not a one-size-fits-all minReplicas parameter for every model.

When to Apply This

This approach is especially well-suited to:

• A single GPU cluster hosting many different foundation models;
• Large model weights where load time after scale-out significantly exceeds Pod scheduling time;
• The need to Scale-to-Zero or run long-tail models with minimal footprint;
• Significant network distance between object storage and GPU nodes;
• Frequent model version updates requiring controlled cache pre-warming and invalidation;
• Clear TTFT / availability SLOs without wanting to keep large numbers of idle GPUs permanently.

If you have a single fixed model, stable traffic, and all GPUs are always resident, the benefits of cold-start governance are limited — optimizing runtime throughput and cost is a better use of effort.

Common Pitfalls

Pitfall 1: Increasing minReplicas is the solution to cold starts. It only reduces the probability that a user hits a cold start. It doesn’t solve weight loading during node rebuilds, failovers, model version switches, or burst scale-outs.

Pitfall 2: Only optimizing the container image. In LLM serving, the container image is usually far smaller than the model weights. Image pull optimization matters, but don’t treat a few-hundred-MB image problem as a tens-of-GB weight problem.

Pitfall 3: Shared network storage equals local cache. A shared disk avoids repeated downloads from the internet, but it doesn’t provide the data path of local NVMe. During concurrent scale-outs, network storage itself can become a new hotspot.

Pitfall 4: Caching without version governance. If the model URI, revision, tokenizer, config, and weight file versions aren’t managed together, you can easily hit a cache with mismatched versions. The cache key should at minimum include the model identifier and an immutable revision.

Pitfall 5: Readiness too early. Process startup doesn’t mean the model is servable. Readiness should only succeed after weight loading, runtime initialization, and necessary warmup are complete; otherwise, traffic gets routed to an instance that’s “alive but can’t infer yet.”

Pitfall 6: Exposing dev-mode Sleep/Wake endpoints directly. Sleep/Wake is a lifecycle control capability, not a business API. In production, it must sit behind an internal control plane with restricted callers.

Pre-Launch Checklist

Before going live, at least verify the following:

• Model weights use immutable version numbers or revisions;
• GPU node NVMe capacity has high-water marks and eviction policies;
• Local Model Cache hit/miss can be measured;
• Fallback paths on cache miss are rate-limited;
• Model loading concurrency has been load-tested against CPU, network, and object storage;
• Readiness covers full warmup;
• Warm Pool has a maximum size and maximum idle time;
• The scheduler prioritizes cache-hit nodes;
• Cold-start stages can be pinpointed to specific time segments;
• New model releases support pre-warming node caches;
• Cache files have checksum / integrity validation;
• Automatic rollback is possible on OOM, download interruption, or version mismatch.

FAQ

Why can’t LLM cold start be solved just by increasing minReplicas?

Because minReplicas only keeps more resident replicas at runtime. As long as node failures, version switches, burst scale-outs, or first-time deployments of new models occur, the cost of fetching and loading weights remains. Real governance requires addressing model locality, loading paths, and temperature tiers together.

What is the fundamental difference between local NVMe cache and a shared network disk?

The advantage of local NVMe isn’t “the file exists on disk” — it’s that the data is already co-located on the target GPU node. A shared network disk still goes through the network and shared storage layer, which can suffer bandwidth contention during large-scale concurrent loads.

Can vLLM Sleep Mode be used directly as a production Warm Pool?

You can borrow its GPU memory release and fast recovery mechanisms, but in production, an internal control plane should wrap it rather than exposing dev-mode endpoints directly. You also need to handle replica state, concurrent wake-ups, failure recovery, and resource quotas.

References

1. USENIX OSDI 2024 — ServerlessLLM: Low-Latency Serverless Inference for Large Language Models: https://www.usenix.org/conference/osdi24/presentation/fu
2. KServe — Local Model Cache: https://kserve.github.io/website/docs/next/model-serving/generative-inference/modelcache/localmodel
3. vLLM — Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/v0.18.0/models/extensions/runai_model_streamer/
4. vLLM — Sleep Mode: https://docs.vllm.ai/en/v0.13.0/features/sleep_mode/