Why is LLM inference more sensitive to cold starts than typical serverless functions?

Because cold starts for LLMs involve more than just container launch; they include image pulling, dependency initialization, weight downloading, weight deserialization, GPU memory allocation, and inference engine warmup. The larger the model, the more the first-token latency is dominated by weight loading.

Is a larger warm pool always better?

No. A warm pool essentially trades persistent GPU cost for lower TTFT. It should be sized based on business SLOs, traffic peaks, model popularity, and cold-start budget, not by simply keeping all models perpetually warm.

Is scale-to-zero suitable for all LLM services?

No. Scale-to-zero is ideal for low-frequency, asynchronous, or queueable tasks. Strongly interactive, low-latency, and high-value tenant services typically require minimum replicas, pre-warmed replicas, or tiered model caching.

LLM Serverless GPU Cold Start in Production: Balancing Cost and TTFT with Weight Prefetching, Warm Pools, and Scale-to-Zero

Background: Cold Start Is More Than Just “Slow Startup”

Turning LLM inference into a Serverless GPU model is superficially appealing: scale to zero when there’s no traffic, spin up GPU workers when requests arrive, and pay per request or per active duration. For internal tools, low-frequency agents, batch summarization, long-tail fine-tuned models, and experimental models, this pattern can significantly reduce idle GPU costs.

LLM Serverless GPU Cold Start in Production: Balancing Cost and TTFT with Weight Prefetching, Warm Pools, and Scale-to-Zero

The problem is that LLM cold starts are fundamentally different from typical web function cold starts. A regular function cold start usually involves container startup, dependency loading, and connection initialization. An LLM cold start must also handle:

Reading model weights from remote object storage, image layers, or local cache;
Checkpoint deserialization and tensor shape parsing;
Moving weights from CPU memory, NVMe, or network storage to GPU memory;
Inference engine initialization: CUDA graph setup, KV cache pre-allocation, tokenizer and serving runtime;
After the first request enters the queue, waiting for the worker to become ready before starting prefill and emitting the first token.

Therefore, the cold start ultimately manifests as TTFT (Time To First Token) . What the user perceives isn’t “the container has started,” but “when will the first part of the answer appear?” If this time jumps from a few hundred milliseconds to tens of seconds, even if TPOT is excellent, the interactive experience is already a failure.

The ServerlessLLM paper breaks this down clearly: LLM checkpoints are large, and cold starts are amplified by remote downloads, loading, memory allocation, and tensor movement. The paper proposes multi-tier checkpoint loading, locality-aware scheduling, and live migration to reduce startup latency. Production systems don’t have to replicate the paper’s system directly, but it highlights a key fact: The core of LLM cold start management is weight locality and shortening the startup path, not just autoscaling parameters.

Core Principles: The Cold Start Path from Request to First Token

A cold start request typically goes through the following path:

request arrives → gateway / external queue → scheduler decides model placement
→ provision GPU worker → pull image / mount volume → load tokenizer and runtime
→ fetch or locate model weights → deserialize and copy weights to GPU memory
→ initialize inference engine → prefill first request → emit first token

In this chain, the most underestimated step is model weight loading. Even if the image is already on the node, the weights might still be in remote object storage, a network file system, or a shared volume. For models ranging from tens to hundreds of GB, network bandwidth, NVMe bandwidth, CPU deserialization, PCIe transfer, and GPU memory allocation all impact TTFT.

Therefore, the design goal in production isn’t to “completely eliminate cold starts,” but to break the cold start down into several controllable variables:

Weight Location: Remote object storage, local NVMe, host memory, GPU memory;
Startup Phase: Image startup, runtime initialization, weight loading, engine warmup;
Scheduling Decision: Whether to select a node with cached weights, whether to queue for a warm worker;
Cost Boundary: How many warm workers to retain, which models allow scale-to-zero;
User Experience: TTFT upper bound, queuing prompts, asynchronous degradation, and timeout strategies.

Engineering Implementation: Building a Control Plane for Cold Start Management

1. Model Tiering: First, Distinguish Hot, Warm, and Cold Models

Don’t put all models under the same Serverless strategy. A practical tiering approach is as follows:

Model Tier	Typical Traffic	Recommended Strategy	Goal
Hot Model	High-frequency online interaction	Minimum replicas + autoscaling	Ensure low TTFT
Warm Model	Periodic access, clear business hours	Small warm pool + pre-warming schedule	Control tail latency
Cold Model	Long-tail, low-frequency, experimental	Scale-to-zero + queue	Reduce idle costs
Large Model	Large weights, slow startup	Local weight cache + locality scheduling	Avoid repeated remote downloads

This step is crucial. Many teams initially treat Serverless as a universal cost-saving switch, scaling even strongly interactive entry points to zero. The result is saving GPU idle costs but losing core user experience.

2. Weight Prefetching: Don’t Let Requests Wait for the Full Download Chain

The goal of weight prefetching is to change from “start moving weights after the request arrives” to “the scheduler already knows which nodes should prepare which models in advance.”

Common practices include:

Building a model popularity table: calculate priority based on recent access frequency, business hours, tenant tier, and model size;
Caching frequently used checkpoints on the node’s local NVMe to avoid downloading from object storage every time;
Prefetching weights to the target node before scaling up, rather than pulling them after the worker starts;
Using sharded weights and parallel reads for large models to avoid single-threaded sequential loading;
Setting a short TTL for canary models to prevent experimental models from occupying local cache long-term.

Modal’s cold start documentation also emphasizes completing initialization work that can be saved to disk (e.g., downloading model weights) as early as possible, moving it out of the first request path. It also notes that entering the warm phase doesn’t eliminate initialization costs; it just shifts them to the pre-warming phase.

3. Warm Pool: Trading a Small Persistent Footprint for Stable TTFT

A Warm Pool isn’t about “keeping all model replicas forever,” but about retaining a small number of ready workers to handle the first wave of burst requests.

When designing a Warm Pool, clearly define these 4 parameters:

model: qwen-xxb-instruct
min_warm_replicas: 1
max_warm_replicas: 4
warm_ttl_seconds: 900
cold_start_budget_ms: 8000

The meaning behind these parameters:

min_warm_replicas: Actively replenish warm workers if the count drops below this value;
max_warm_replicas: Prevent over-warming and cap costs;
warm_ttl_seconds: How long a model can remain idle before it’s allowed to cool down;
cold_start_budget_ms: If a cold start is expected to exceed this budget, the request should be queued, degraded, or routed to a shared model.

BentoML’s autoscaling documentation mentions that scale-to-zero can reduce replicas to zero when the service is idle, and upon a new request, an external queue waits for the service to scale up. This mechanism is suitable for low-frequency tasks, but for strongly interactive LLMs, it must be combined with a cold start budget and a queuing experience.

4. Scale-to-Zero: Not a Default, but a Tiered Strategy

The use cases for Scale-to-Zero are clear:

Internal low-frequency tools;
Asynchronous batch processing tasks;
Non-real-time generation where users can accept waiting;
Long-tail models, experimental models, and temporary tenant models;
Scenarios where results are delivered via queue callbacks, Webhooks, or task status pages.

The unsuitable scenarios are also clear:

Chatbots requiring very fast first-screen response;
Voice agents, real-time collaboration, and coding agents;
High-value tenant SLAs;
Unpredictable business peak traffic where users cannot wait;
Large model weight loading times far exceeding acceptable user latency.

The correct approach is to place Scale-to-Zero within a policy engine, not hardcode it into deployment configurations. For example:

policies:
  - match:
      tenant_tier: enterprise
      model_class: interactive
    scaling:
      min_replicas: 1
      warm_pool: enabled
      scale_to_zero: false
  - match:
      tenant_tier: free
      model_class: async_batch
    scaling:
      min_replicas: 0
      warm_pool: disabled
      scale_to_zero: true

5. Locality-Aware Scheduling: Schedule to Nodes That “Have the Weights”

If Node A already has a model’s weights cached, and Node B needs to download them remotely, the scheduler shouldn’t just look at GPU availability; it should also consider checkpoint locality.

A practical scheduling score could include:

score = gpu_available_score + local_weight_cache_score + warm_worker_score
        - queue_delay_penalty - eviction_risk_penalty

The key here isn’t the formula itself, but incorporating “where the weights are” into scheduling. One of the core ideas of ServerlessLLM is to leverage the multi-tier storage on GPU servers and perform startup-time-optimized scheduling based on checkpoint locality. Production systems can simplify the implementation: start with local NVMe caching and model popularity scoring, then gradually expand to host memory caching, sharded prefetching, and migration.

6. External Queue: Turning Cold Start Waiting into Controllable Waiting

Frameworks like Ray Serve and BentoML all emphasize the relationship between concurrency, queuing, and autoscaling. Under LLM cold starts, the queue isn’t just a traffic-shaping tool; it’s also a user experience management tool.

It’s recommended to log the following at the gateway layer:

Request enqueue time;
Selected model and version;
Whether a warm worker was hit;
Whether a cold start was triggered;
Weight loading start/end times;
Worker ready time;
TTFT;
Whether the request was degraded due to cold start timeout.

This way, when troubleshooting, you can answer: Was the slowdown due to no warm worker for the model, a weight cache miss, or a slow engine warmup even after the GPU worker started?

Applicable Scenarios

This approach is suitable for the following scenarios:

Internal multi-model platforms: Many models with vastly different access frequencies; it’s impossible to keep all of them resident on GPUs.
Long-tail fine-tuned model serving: Each tenant has its own model or adapter, but access patterns are uneven.
Asynchronous generation tasks: Summarization, reports, batch rewriting, offline annotation, etc., can be queued.
Experimental model platforms: Low-cost trial runs for new models before full deployment.
Multi-region cost optimization: Allow scale-to-zero in low-traffic regions, retain warm pools in high-traffic regions.

This approach should not be blindly applied to all online entry points. For core interactive services, the cost of minimum replicas and pre-warming is usually a necessary expense.

Common Misconceptions

Misconception 1: Only Looking at GPU Utilization, Ignoring TTFT

High GPU utilization doesn’t mean a good user experience. During a cold start, the GPU might not even be in the generation phase yet, while the user is already waiting. LLM Serverless requires putting TTFT, cold_start_rate, and warm_hit_rate on the core dashboard.

Misconception 2: Bundling Weights into the Image Solves Everything

Putting weights into the image can reduce runtime downloads, but it leads to huge images, slow deployments, slow rollbacks, and node cache pollution. A more robust approach is to separate the image, weights, and runtime configuration, then control the loading path through local caching and prefetching strategies.

Misconception 3: Warm Pools Never Save Money

Warm pools do have a persistent cost, but they can be limited to a small number of hot model replicas, controlled by TTL, popularity, and tenant tier. The problem isn’t the warm pool itself, but a warm pool without boundaries.

Misconception 4: Scale-to-Zero Always Hurts the Experience

For asynchronous and low-frequency tasks, scale-to-zero is a reasonable choice. The key is to indicate the queuing status at the product layer, set a cold start budget at the engineering layer, and avoid reloading weights from remote storage every time at the scheduling layer.

Go-Live Checklist

Before going live, at least check the following items:

Conclusion

The challenge of LLM Serverless GPU isn’t “whether it can scale to zero,” but “when it scales back from zero, how long will the user wait, why, and is it worth the wait?”

Production systems need to break the cold start into an observable, schedulable, and degradable pipeline: weight prefetching solves the loading path, warm pools safeguard the interactive experience, scale-to-zero manages idle costs, and TTFT metrics bring user perception back into engineering decision-making.

When these mechanisms form a closed loop, Serverless GPU becomes more than just a cost-saving button; it becomes an inference infrastructure that can be finely tuned by model, tenant, and business scenario.

References

Modal Docs: Cold start performance — https://modal.com/docs/guide/cold-start
BentoML Docs: Concurrency and autoscaling / Scale-to-Zero — https://docs.bentoml.com/en/latest/scale-with-bentocloud/scaling/autoscaling.html
Ray Serve Docs: Autoscaling — https://docs.ray.io/en/latest/serve/autoscaling-guide.html
Runpod Docs: Serverless Workers Overview — https://docs.runpod.io/serverless/workers/overview
ServerlessLLM: Low-Latency Serverless Inference for Large Language Models — https://arxiv.org/abs/2401.14351