Serverless GPU Inference Cold Start Mitigation: Reducing First-Token Latency with Warm Pools and Staged Pre-Warming

Background: Serverless Saves Money, but Cold Starts Amplify First-Token Latency

LLM inference teams are increasingly adopting serverless GPU or elastic GPU pools to reduce idle costs during low-traffic periods: scale down when there are no requests, and spin up GPU workers when traffic spikes. The problem is that cold starts for LLM services are not the same as for typical web functions. A standard function might just spin up a container and load dependencies; LLM inference involves image startup, model weight downloads, moving weights from disk to CPU memory to GPU memory, runtime initialization, tokenizer or inference service warmup, and queuing for the first request.

Modal’s cold start documentation breaks down the additional latency into two categories: the queue time a request spends waiting for a warm container, and the initialization work a new container must complete before processing its first request. The docs also explicitly note that if large model weights are downloaded from a remote source at startup, startup time can jump from seconds to minutes, while pre-downloading or embedding them in the image/volume can significantly reduce startup time. Ray Serve’s autoscaling documentation also emphasizes that autoscaling decisions are based on the request queue, and min_replicas=0 is suitable for scenarios with long periods of no traffic where you can tolerate scaling tail latency. These two facts together mean that cold start mitigation cannot rely solely on “enabling autoscaling”; the model lifecycle must be managed separately.

This article does not discuss Continuous Batching, PagedAttention, Prefix Caching, or LoRA adapter caching. Instead, it addresses a more platform-level problem: when a model, tenant, or version is revived from a cold state, how do you prevent the first request from being crippled by model loading, without reserving all GPUs as hot instances?

Where Does the Cold Start Actually Slow Down?

For LLM serverless inference, a cold start can be broken down into six stages:

Stage	Typical Problem	Key Mitigation Action
Scheduling	No warm worker available for the request; must wait for a new one	Warm pool, buffer workers, predictive pre-warming
Container	Slow image pull, dependency loading, Python imports	Pre-built images, dependency trimming, memory snapshots
Weights	Model weights read from object storage or remote repository	Local SSD cache, volumes, sharded parallel reads
GPU	Weights moved to VRAM, CUDA runtime initialization	Pre-loading, VRAM reservation, tiered eviction
Service	Tokenizer, server, routing, health checks not yet ready	Readiness gates, warmup requests, shadow probing
First Request	First real request bears the cost of compilation, kernels, cache initialization	Synthetic warmup, first-request isolation, cold/warm metric splitting

Many teams focus only on “container startup time,” but the larger cost for LLM inference is often model weight loading and GPU VRAM restoration. The ParaServe paper points out that the cold start latency of serverless LLM serving is affected by the size of the large model, with fetching the model from remote storage being a primary bottleneck; it reduces cold start time by parallelizing model parameter fetching, loading, and runtime initialization. Tangram approaches the problem from another angle: leveraging GPU memory reuse and GPU affinity to minimize parameter movement when a cold model is restarted.

Core Principle: Treat the Cold Start as a “Model Lifecycle Pipeline”

The core of cold start mitigation is not a single parameter, but defining the process from “not serviceable” to “stably serviceable” as a clear state machine:

COLD -> IMAGE_READY -> WEIGHTS_ON_DISK -> WEIGHTS_IN_CPU_MEMORY
     -> WEIGHTS_ON_GPU -> RUNTIME_READY -> WARMED -> SERVING
     -> IDLE -> EVICTING -> COLD

Each state must have observable metrics, timeout thresholds, and fallback actions. For example, a WEIGHTS_ON_DISK timeout usually indicates an issue with object storage, image layers, or local cache; a WEIGHTS_ON_GPU timeout might be due to insufficient GPU memory, a slow eviction policy, or too many concurrent loads; a failure between RUNTIME_READY and WARMED often means the health check passed too early, forcing real requests to bear the warmup cost.

Once the cold start is broken down into a state machine, the platform can do three things:

1. Shift cold start costs forward: Complete model downloads, weight loading, and warmup during deployment, scaling, or traffic prediction phases.
2. Isolate cold start costs: Don’t let the first request from a real user bear all initialization costs.
3. Make cold start costs explicit: Separate statistics for cold TTFT, warm TTFT, model load time, and queue wait time.

Warm Pool: Not Keeping GPUs On Forever, but Retaining “Quickly Takeoverable” Hot Capacity

A common misconception about warm pools is “just increase min_replicas.” That’s the crudest approach. A real warm pool needs to answer four questions: which models are worth keeping resident, at what lifecycle layer to retain them, what events trigger pre-warming, and when to evict.

Tier	Strategy	Applicable To
Hot	Resident GPU, fully warmed	High-frequency primary models, core paying tenants, low-latency APIs
Warm	Container or weights resident, but not necessarily occupying full GPU	Medium-frequency models, peak work hours, predictable batch tasks
Cold	Completely on-demand loading	Long-tail models, low-frequency experimental models, internal offline tasks

This is not a simple QPS ranking; it also depends on model size, loading time, VRAM usage, SLO level, tenant value, and the presence of periodic traffic.

Not all models need to reach WEIGHTS_ON_GPU. Some models only need to reach WEIGHTS_ON_DISK, avoiding remote downloads without occupying GPU VRAM. Some high-value models need to reach RUNTIME_READY or WARMED to ensure immediate service when a real request arrives.

warm_pool_policy:
  hot_models:
    target_state: WARMED
    min_workers: 2
    scaledown_window_seconds: 1800
  warm_models:
    target_state: WEIGHTS_ON_DISK
    min_workers: 0
    prefetch_window_minutes: 15
  cold_models:
    target_state: COLD
    min_workers: 0
    max_cold_start_seconds: 120

Pre-warming signals can come from deployment events, historical traffic, business activities, and real-time queues. Eviction should not be based solely on idle time; it should also consider model size, cold start cost, tenant tier, current GPU pool pressure, and the probability of a recently evicted model being requested again.

Research like Tangram and CrossPool points to a common fact: one of the core bottlenecks in LLM cold starts and multi-model serving is the management of “static weights” and “dynamic request state” in GPU memory. Engineering implementations don’t have to copy these systems exactly, but they should absorb their ideas: manage weights, KV cache, worker lifecycle, and scheduling affinity separately.

Staged Pre-Warming: Don’t Let the First Real Request Do a System Health Check

A deployable serverless GPU inference service should have at least four levels of pre-warming.

Build-Time Pre-Warming

Build time doesn’t involve GPUs, but aims to remove slow I/O from runtime: freeze the base image layer, pre-download tokenizers and configs, solidify unchanging dependencies into the image or volume, and record the model artifact digest to prevent runtime drift.

Startup Pre-Warming

Startup time should advance the container from “startup complete” to “serviceable.”

class LLMWorker:
    def __init__(self, model_path: str):
        self.model_path = model_path
        self.ready = False
        self.model = None

    def startup(self):
        self.tokenizer = load_tokenizer(self.model_path)
        self.model = load_model_from_local_cache(self.model_path)
        self.model.to("cuda")
        self.model.eval()
        self.warmup()
        self.ready = True

    def warmup(self):
        prompt = "warmup"
        _ = self.model.generate(prompt, max_new_tokens=1)

Note that ready = True must be set after warmup. Otherwise, the service discovery layer will route real traffic to a worker that is “alive but not warmed.”

Release Pre-Warming

New model versions first enter shadow warmup, not receiving real traffic. After warmup passes, shift 1% of traffic. Simultaneously compare cold TTFT, warm TTFT, error rate, and GPU memory peak. When rolling back, retain the previous version’s warm workers for a period to avoid a cold start on rollback.

Traffic Pre-Warming

Traffic time must handle bursts: when the pending queue reaches a threshold, not only scale up replicas but also pre-load the corresponding model. Use scheduled warmup for periodic traffic, set higher warm priority for high-value tenants, and use local weight caching for long-tail models instead of resident GPUs.

Engineering Architecture

It is recommended to split cold start mitigation into five components:

1. Traffic Classifier: Identifies model, tenant, SLO, and request type.
2. Warm Pool Manager: Decides which models are kept at which state.
3. Artifact Cache Manager: Manages image layers, model weights, local SSDs, and volumes.
4. GPU Residency Controller: Decides which weights stay on the GPU and which are demoted to CPU or disk.
5. Readiness & Warmup Gate: Only allows real traffic after warmup passes.

Request -> Gateway -> Traffic Classifier -> Warm Pool Manager
       -> Worker Pool -> Readiness Gate -> LLM Runtime

Artifact Store -> Local Cache -> CPU Memory -> GPU Memory

The key to this architecture is: the gateway should not only know how many workers exist, and the scheduler should not only know if a GPU is idle. They both need to know if the model is hot, where the weights are, and what the current cold start cost is.

Metrics: Must Distinguish Cold from Warm

If you only look at average latency, the cold start problem will be masked. It is recommended to at least break out the following metrics:

Metric	Meaning	Purpose
cold_start_rate	Proportion of requests hitting a cold worker	Assess if the warm pool is sufficient
cold_ttft_p95	P95 first-token latency for cold requests	Measure the worst user experience
warm_ttft_p95	P95 first-token latency for warm requests	Assess model service quality excluding cold starts
model_load_time	Time to load weights	Identify issues with remote storage, disk, CPU/GPU transfer
queue_wait_time	Time a request waits for a worker	Determine if scaling is lagging
warmup_failure_rate	Proportion of warmup failures	Prevent unprepared workers from receiving traffic
warm_pool_cost	Idle cost of warm workers	Evaluate the cost-latency trade-off
eviction_miss_rate	Proportion of recently evicted models requested again	Tune the eviction policy

In production dashboards, break these metrics down by model, tenant, version, GPU pool, and region. Cold starts are usually not a global average problem but a problem for a few models or tenants hit repeatedly during specific time periods.

Applicable Scenarios

This approach is suitable for:

• Multi-model platforms where only a few models are high-frequency and the rest are long-tail.
• Enterprise customers using independent model versions or dedicated LoRA/adapters.
• Inference platforms that allow scale-to-zero and want to reduce GPU idle costs.
• Large model volumes where the first load significantly impacts TTFT.
• Obvious periodic traffic patterns, such as work hours, event times, or batch processing windows.
• When releasing new models, the first batch of users often experiences noticeably slow requests.

The scenarios where it is not suitable are also clear: if the service requires 24/7 low latency, has few models, and stable traffic, simply fixing the number of replicas might be simpler. The value of a warm pool lies in handling the middle ground where “cost doesn’t allow everything to be hot, but the experience doesn’t allow everything to be cold.”

Common Misconceptions

Misconception 1: Autoscaling Alone Solves Cold Starts

Autoscaling solves insufficient capacity, not necessarily slow model loading. Ray Serve can scale replicas based on the queue, but if a replica must pull the model from a remote source after starting, the user will still wait.

Misconception 2: A Warm Pool Means All Models Are in the GPU

GPU VRAM is the most expensive resource. Long-tail models are better suited for WEIGHTS_ON_DISK or WEIGHTS_IN_CPU_MEMORY; only high-frequency models need WEIGHTS_ON_GPU or WARMED.

Misconception 3: Passing Health Checks Means the Model is Serviceable

A container being alive does not mean the model is warmed. Health checks should at least distinguish between process_alive, runtime_ready, model_loaded, and warmup_passed.

Misconception 4: P95 Latency is Enough

Cold starts usually affect a small proportion of requests, but those requests have a terrible experience. Cold path metrics must be tracked separately; otherwise, averages will mislead decision-making.

Go-Live Checklist

• Does the model weight have a fixed digest?
• Does the image contain necessary dependencies to avoid runtime installation?
• Can the tokenizer, config, and weight path be read from a local cache?
• Does the warmup request cover the real inference path?
• Does the readiness gate only allow traffic after warmup?
• Can old and new versions retain warm workers simultaneously?
• Are the Hot/Warm/Cold model tiers defined?
• Are the target state and maximum cost set for each tier?
• Is warm priority set for high-value tenants?
• Is there a GPU VRAM watermark and eviction policy?
• Is the number of concurrent model loads limited to avoid an I/O storm?
• Are cold TTFT and warm TTFT distinguished?
• Are model load time, queue wait time, and startup pending time recorded?
• Is there a warmup failure alert?
• Is the warm pool idle cost recorded?

References

• Modal Docs: Cold start performance — https://modal.com/docs/guide/cold-start
• Ray Serve Autoscaling Guide — https://docs.ray.io/en/latest/serve/autoscaling-guide.html
• ParaServe: Towards Swift Serverless LLM Cold Starts — https://arxiv.org/abs/2502.15524
• Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity — https://arxiv.org/abs/2512.01357
• Serverless Cold Starts and Where to Find Them — https://arxiv.org/abs/2410.06145
• CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation — https://arxiv.org/abs/2606.24506