What is the difference between LoRA adapter caching and regular model caching?

Regular model caching typically revolves around full weights or inference instances. LoRA adapter caching focuses on small delta weights on top of a shared base model, with emphasis on managing heat, loading, eviction, and version consistency per adapter.

Can I just preload all LoRA adapters onto the GPU?

For small scale, yes. But as the number of adapters, ranks, tenants, and versions grows, GPU memory becomes dominated by adapters, impacting KV cache, batch size, and tail latency. Heat tiering and eviction policies become necessary.

Is dynamic LoRA adapter loading suitable for direct exposure to external tenants?

No. In production, dynamic loading should be managed through a controlled registry, signature verification, canary admission, and tenant authorization to prevent arbitrary path loading and unverified weights from entering the inference process.

Should LoRA adapter caching be handled by the gateway or the inference engine?

Both. The inference engine handles actual loading, execution, and internal memory management. The gateway or control plane handles tenant policies, version admission, warm-up plans, eviction priorities, and auditing. Don't cram all business logic into inference engine parameters.

LLM LoRA Adapter Cache in Production: Stabilizing Multi-Tenant Fine-Tuning Services with Heat Tiering, Cold-Start Budgets, and Eviction Policies

The Problem: LoRA Makes Custom Models Lightweight, But Doesn’t Eliminate Service Governance

The value of LoRA is clear: keep the base model unchanged, train only a small set of adapter delta weights for each business scenario—customer service, finance, code, legal, operations—and you get a customized model. This is a huge saving for training and storage. For online inference, however, the problem is merely shifted.

When a platform has only a few adapters, the simplest approach is to load them all at startup. But in a multi-tenant scenario, adapters quickly multiply to dozens, hundreds, or more. Each tenant may have production, canary, rollback, and A/B test versions; each business line may further split by language, industry, or region. At this point, the bottleneck is no longer “can we load LoRA?” but rather:

Which adapters should reside permanently on the GPU, and which should only be in CPU memory or object storage?
When a cold adapter is requested, how much additional TTFT (Time To First Token) is acceptable?
Will in-flight requests be affected when an adapter is evicted?
How do we ensure version consistency when an adapter with the same name is updated?
Will hot tenants evict adapters from cold but high-priority tenants?
Are adapter files signed, format-validated, and checked for base model compatibility?

Therefore, the core of LoRA productionization is not re-explaining low-rank matrices, but building an Adapter Cache management layer.

Core Principles: Treat Adapters as Independent Resources, Not Appendages of Model Parameters

A LoRA request can be broken down into three resource types:

Resource Type	Description	Load Cost	Lifecycle
Base Model	Full large model weights, resident on GPU or managed by inference engine	Highest	Longest
Adapter Weight	Much smaller than base model, many versions, highly skewed access distribution	Medium	Changes with heat
Request Runtime State	KV Cache, queue position, sampling parameters, streaming state, request accounting	Low	Request-level

In a production system, the Adapter Cache needs at least three tiers:

1. GPU Hot Tier

The GPU hot tier holds currently high-frequency adapters. The goal is to prevent hot requests from triggering adapter loads, thus stabilizing TTFT and P95/P99 latency.

However, the GPU hot tier cannot expand indefinitely. Adapter memory competes with KV Cache, batch size, context length, and concurrent requests. A common mistake is to treat GPU utilization as the only metric—if adapters consume too much, the GPU may appear busy, but throughput and tail latency will degrade.

2. Host Warm Tier

The Host Warm Tier typically resides in CPU memory or local NVMe. It holds adapters that may warm up again soon, reducing the cost of downloading, decompressing, validating, and format-converting from remote object storage.

This tier is suitable for adapters that are “not very hot, but can’t be too slow”—for example, enterprise key accounts, whitelisted tenants, business lines about to enter an active period, or new versions in canary release.

3. Artifact Cold Tier

The Cold Tier is object storage, a model registry, or an artifact repository. It must retain full versions, signatures, metadata, training lineage, and compatibility information. Any adapter moving from the Cold Tier to production must pass an admission check, not just a path existence check.

Engineering Implementation: Start with an Adapter Registry, Not Inference Engine Parameters

Establish an Adapter Registry

The Adapter Registry is the control plane for the LoRA production system. Each adapter should have at least these fields:

adapter_id: "tenant-a-support-v3"
base_model: "llama-3.1-8b-instruct"
version: "2026-07-09.1"
rank: 16
format: "peft"
artifact_uri: "s3://model-artifacts/lora/tenant-a-support/v3"
sha256: "..."
status: "active"
tenant_id: "tenant-a"
priority: "gold"
compatibility:
  tokenizer_revision: "..."
  target_modules: ["q_proj", "v_proj"]
  max_lora_rank: 32
release:
  canary_percent: 10
  rollback_to: "2026-06-30.2"

The value of this table goes beyond querying paths. It determines whether an adapter can be loaded, which base model it loads onto, whether canary is allowed, whether eviction is permitted, and whether a minimum hot replica must be retained.

Design a Heat Score, Not a Simple LRU

A simple LRU tends to keep “low-value adapters that just had a burst of access” on the GPU while evicting “critical adapters with low frequency but high SLA.” A more reasonable approach is to use a weighted heat score:

adapter_cache_policy:
  gpu_slots_per_base_model: 64
  host_slots_per_base_model: 2048
  eviction_policy: "weighted-lru"
  hot_score_formula: |
    0.45 * qps_10m +
    0.25 * ttft_penalty_if_cold +
    0.20 * tenant_priority +
    0.10 * scheduled_warmup_score
  never_evict_when:
    - status == "canary" and canary_percent > 0
    - tenant_priority == "platinum"
  demote_when:
    - no_request_for_minutes > 60
    - error_rate_after_load > 0.02

The key here is to combine request frequency, cold-start penalty, tenant priority, and business warm-up signals. Otherwise, the cache policy will only serve average throughput, not business goals.

Set a Cold-Start Budget

LoRA cold start includes: parsing the adapter ID, reading the registry, downloading or locating the file, verifying the signature, loading to the host, moving to the GPU, and updating the inference engine’s internal mapping. Different frameworks support these actions differently, but the business side should unify them into a cold-start budget.

It is recommended to set at least three types of budgets:

Budget Type	Use Case	Strategy
Interactive Request Budget	Customer service conversations, code completion, online Q&A	Cold adapters should not enter the main path directly; can return a queue, degrade to base model, or pre-warm
Background Task Budget	Offline summarization, batch quality checks, data annotation	Longer load times acceptable, but limit concurrent cold loads
Canary Release Budget	New adapter before user traffic	Use shadow requests and a golden set for warm-up first, avoiding a full cold start on the first real access

Put Dynamic Loading in a Controlled Channel

vLLM supports LoRA per-request usage and dynamic adapter loading via API endpoints or resolver plugins; TGI supports declaring multiple LoRA adapters at startup via environment variables and specifying the adapter in the request. They provide engineering capability, but production governance cannot rely solely on inference engine switches.

The production system should encapsulate dynamic loading as an internal control plane:

Business Request → LLM Gateway → Adapter Registry Query
    → Adapter Cache Manager Decision
    → Inference Engine load / pin / use
    → After request completes: update heat and cost ledger

Do not let external tenants pass adapter paths directly, and do not allow unsigned adapters to enter the inference process. In particular, remote download capabilities should only be open to trusted environments, and should be accompanied by allowlists, signature verification, size limits, format checks, and audit logs.

Applicable Scenarios

This solution is suitable for:

Enterprise SaaS maintaining custom customer service models for each client;
Internal AI platforms providing multiple SFT/LoRA versions on the same base model for different teams;
Code assistants loading different adapters by language, project, or team style;
Content production platforms maintaining adapters by brand tone, market region, or compliance policy;
Multi-tenant inference platforms that want to share base model GPU memory while providing custom model entry points.

If your system has only one or two adapters with very low release frequency, you don’t need a complex caching layer from the start. Just use fixed loading, fixed release, and fixed rollback. Complexity should come from real growth, not architectural imagination.

Common Misconceptions

Misconception 1: LoRA is small, so memory management is unnecessary

A single adapter is small, but the set of adapters is not. In multi-tenant systems, the number of adapters, versions, and rank differences will continue to grow. An overly large GPU hot tier will squeeze KV Cache and batch space.

Misconception 2: LRU is sufficient for eviction

LRU only represents recent access, not business priority, cold-start cost, or SLA. Production environments must incorporate tenant tier, canary status, activity warm-up, error rate, and load cost into the policy.

Misconception 3: Dynamic loading equals production readiness

Dynamic loading is a capability, not governance. Without signature verification, version locking, path isolation, auditing, and rollback mechanisms, dynamic loading becomes a risk entry point.

Misconception 4: Only monitor base model metrics

LoRA services must be monitored per adapter dimension. Otherwise, you only see overall GPU utilization, but miss cold-start counts, load failure rates, hit rates, and version drift for individual adapters.

Pre-Launch Checklist

Before going live, check each item:

#	Check Item	Description
1	Registry Completeness	Is each adapter bound to base model, rank, format, target modules, version, and sha256?
2	Load Security	Is arbitrary path loading prohibited? Are only trusted repositories and signed artifacts allowed?
3	Cache Tiering	Are GPU hot, host warm, and artifact cold tiers distinguished?
4	Eviction Protection	Do canary, whitelist, and high-SLA adapters have minimum residency or warm-up policies?
5	Cold-Start Budget	Do interactive requests, background tasks, and canary requests use different budgets?
6	Version Consistency	When an adapter name is updated, do in-flight requests continue with the old version, or is in-place replacement allowed?
7	Metric Dimensions	Are hit rates, load times, load failure rates, TTFT, P95/P99 output per base_model, adapter_id, tenant_id, and adapter_version?
8	Rollback Path	Can an adapter roll back to the previous version or degrade to the base model on failure?
9	Stress Test Scenarios	Are hot adapters, cold adapters, burst loads, batch evictions, repository unavailability, and signature failures covered?

Summary

LoRA adapter cache governance is one of the core engineering challenges of multi-tenant fine-tuning services. It is not a sub-problem of GPU memory management, but a control plane system that requires independent design. Key takeaways:

Three-tier cache architecture: GPU Hot → Host Warm → Artifact Cold, with heat-based demotion;
Weighted heat score instead of LRU: Combine request frequency, cold-start penalty, tenant priority, and business warm-up signals;
Classified cold-start budgets: Interactive requests, background tasks, and canary releases use different budgets and degradation strategies;
Controlled loading channel: All dynamic loading must pass registry validation; tenants cannot pass paths directly;
Per-adapter monitoring: From hit rate to P99 latency, drill down to individual adapters.

If your multi-tenant LoRA service is transitioning from “full load of a few adapters” to “on-demand scheduling of dozens or hundreds of adapters,” now is the best time to establish this governance system.