Are Encoder Cache and KV Cache the same thing?

No. Encoder Cache stores embeddings output by vision or other media encoders, aiming to skip repeated media encoding; KV Cache stores key/value states of the language model's attention layers, aiming to reduce repeated prefill. Their keys, capacity, invalidation, and observability should all be designed independently.

Is it necessary to split the Encoder into a separate Worker for multimodal services?

Not necessarily. When concurrency is low, images are few, or vision encoding is not the bottleneck, co-located deployment is simpler. Only when the encoding queue, Encoder GPU utilization, or vision-stage P95/P99 clearly become bottlenecks is a dedicated Encoder Worker usually worth introducing.

Why can't overall TTFT alone judge multimodal performance?

Because TTFT mixes media download, decode, Processor, vision encoding, embedding transfer, and language model prefill together. Without stage-level metrics, it's easy to misdiagnose a CPU decode bottleneck as a GPU inference problem.

Multimodal LLM Serving in Production: Taming Vision Request Tail Latency with Parallel Media Decode, Encoder Cache, and Encoder Disaggregation

After deploying multimodal LLMs, many teams are surprised to find that even with low GPU utilization, the P95/P99 time-to-first-token (TTFT) for vision requests keeps degrading. The problem often lies not in the language model itself, but in the “invisible pre-processing pipeline” before images or videos enter the model—remote fetching, decoding, resize/normalize, vision encoding, and embedding transfer. This article draws on NVIDIA Dynamo, vLLM, and EPD practices to lay out a practical approach for governing the vision critical path.

Why Multimodal Requests Can’t Be Governed Like “Ordinary LLM Requests”

The online path for pure-text LLMs is relatively clear: requests enter the scheduler, complete prefill, and then proceed to token-by-token decode. Vision language models (VLMs) add a pre-processing pipeline that is easy to overlook:

HTTP / Base64 media
  ↓ fetch / decode / decompress
  ↓ resize / normalize / multimodal processor
  ↓ vision encoder
  ↓ multimodal embeddings
  ↓ LLM prefill + decode

This means a request that looks like “just sending one more image” can actually consume network bandwidth, CPU decode threads, host memory, vision Encoder GPU compute, and language model GPU compute simultaneously. Once image count, resolution, video frame count, and media sources go out of control, P95/P99 TTFT will degrade first, while average throughput may not immediately reveal the problem.

NVIDIA Dynamo’s current multimodal serving documentation breaks this pipeline into several independently optimizable stages, including Parallel Media Decoding, Embedding Cache, Multimodal KV Routing, and Encoder Disaggregation. This article focuses on the parts directly related to the “vision input pre-processing / encoding stage.”

Core Principle: Break Down the Vision Critical Path First, Then Decide on Optimizations

The first step in multimodal performance governance is not changing the architecture, but breaking the critical path into independently measurable stages. Only by seeing where the bottleneck lies can you decide whether to optimize CPU, add caches, or split out the Encoder.

1. Move Image Fetching and Decoding Off the Inference Worker’s Critical Path

Remote image requests typically involve CPU and I/O work such as URL fetch, Base64 decode, and JPEG/PNG/WebP decompression. If all of this happens inside the model Worker, the GPU may be waiting for the CPU to prepare inputs, while the model process also bears unpredictable network latency.

Dynamo’s Parallel Media Decoding moves image fetching, Base64 decoding, and image decompression to a front-end CPU worker pool, which runs concurrently and then hands decoded pixel buffers to the inference backend. The value of this design is not “making the vision Encoder faster,” but moving media work off the GPU Worker’s request threads where it doesn’t belong.

Scenarios where enabling this first makes sense typically include:

Requests frequently carry HTTP/HTTPS images;
Single requests contain multiple images;
High image compression ratios with significant decode overhead;
Backend CPU has become a bottleneck on the request path;
GPU utilization is low, but TTFT remains long and volatile.

From an engineering standpoint, don’t just measure total TTFT. At minimum, record the following metrics independently:

media_fetch_ms
media_decode_ms
mm_processor_ms
vision_encode_ms
embedding_transfer_ms
llm_prefill_ms
tpot_ms

If media_fetch_ms + media_decode_ms already accounts for the bulk of vision request TTFT, address front-end decoding and media sources first, rather than directly adding LLM GPUs.

2. Treat Encoder Cache as an Independent Cache Layer—Don’t Confuse It with KV Cache

For use cases like product images, fixed screenshots, shared flowcharts, or multi-turn Q&A over the same image, request content may change constantly, but the media itself is often repeated. Re-running the vision Encoder every time wastes a lot of duplicate compute.

Dynamo’s Embedding Cache / Encoder Cache uses a CPU-side LRU cache to store vision Encoder outputs, allowing hits to skip vision encoding. vLLM also provides a multimodal Processor Cache and allows stable multi_modal_uuids to avoid re-hashing raw media content every time.

Three types of caches need to be clearly distinguished:

Cache	Cached Object	Primary Savings	Typical Invalidation Conditions
MM Processor Cache	Preprocessed multimodal inputs	CPU work like resize, processor	Processor params, input content changes
Encoder / Embedding Cache	Vision Encoder output embeddings	Vision Encoder GPU compute	Encoder, Processor, input content changes
KV Cache	LLM attention key/value	LLM prefill	Prompt/token prefix changes

If you merge all three into a single “cache hit rate,” online troubleshooting becomes nearly meaningless. We recommend observing hit rate, capacity, eviction count, and stage time saved separately.

Cache keys should not rely solely on image URLs. CDN URLs can remain unchanged while content updates, and the same URL can produce different embeddings under different image Processor or vision Encoder versions. A more robust production key should include at least:

media_content_hash + encoder_model_version + processor_version + processor_parameters

If your business uses stable media IDs, ensure the ID maps to immutable content, or include a content version number in the cache key.

3. Only Do Encoder Disaggregation When the Encoder Is Truly the Bottleneck

Multimodal models typically co-locate vision encoding and language model inference in the same service instance. The advantage is simple deployment; the downside is that when vision requests spike, the Encoder and LLM compete for resources on the same GPUs, and the two stages don’t necessarily have matching resource requirements.

Encoder Disaggregation separates the vision Encoder into dedicated Encode Workers, which produce embeddings that are then passed to downstream LLM Workers. Dynamo currently supports topologies like E/PD and E/P/D, but from a production governance perspective, the key point is that “the Encoder can scale independently and use different hardware”—not splitting for the sake of splitting.

                ┌─ Encode Worker 1 ─┐
Media Decode ───┼─ Encode Worker 2 ─┼──> embeddings ──> LLM Worker Pool
                └─ Encode Worker N ─┘

Signals that favor splitting include:

vision_encode_ms P95/P99 consistently accounts for a significant share of TTFT;
Encoder queues grow while LLM decode GPUs still have headroom;
The traffic ratio between image and text requests fluctuates widely;
Vision Encoder and language model suit different GPU types;
You need independent capacity control for the vision stage to prevent image traffic from slowing down text-only traffic.

Conversely, if your business is just low-concurrency image Q&A with low Encoder utilization, the added RPC, embedding transfer, deployment topology, and failure domains from splitting may outweigh the benefits. Co-located deployment should be the default baseline; separation is a bottleneck-driven optimization.

The research paper “Efficiently Serving Large Multimodal Models Using EPD Disaggregation” also validates the potential of independently provisioning the encoding stage. However, the TTFT and throughput improvements in the paper depend on the model, image count, hardware, and traffic patterns—production environments should not directly copy the paper’s gains. Rely on your own stage-level profiling.

Input Budgets Matter More Than “Backend Hardening”

Multimodal interfaces naturally amplify request costs: a single 512×512 image and 20 high-resolution images are not the same cost; an 8-frame video and a hundreds-of-frames video are not the same load.

vLLM’s multimodal configuration allows limiting the number of media items per prompt and constraining parameters like video frame count, width, and height. In production, these limits should be elevated to API contract level, not hidden engine parameters.

For example, you can establish a tiered budget like this:

multimodal_budget:
  default:
    max_images: 4
    max_image_width: 2048
    max_image_height: 2048
    max_video_frames: 32
  trusted_batch_job:
    max_images: 16
    max_video_frames: 96

What really needs controlling isn’t just “file size,” but also:

Image count;
Total decoded pixel volume;
Video frame count;
Video duration and sampling strategy;
Media download timeout;
Cumulative media bytes per request;
Number of vision tokens produced by the Processor.

These fields should flow into both request logs and cost attribution; otherwise, multimodal services will struggle to explain “why the same API call varies by multiples in latency and GPU cost.”

Media URLs Are Also a Production Boundary

Once you support image_url and video_url, the server actively accesses external addresses. Without a URL policy, this capability introduces SSRF and internal network probing risks.

Dynamo’s current documentation explicitly defines a default URL validation policy: allow HTTPS and data URLs by default, block private/internal and loopback addresses, and only relax restrictions in clearly internal deployment scenarios. Regardless of which serving framework you use, you should establish similar rules:

Reject internal networks and metadata endpoints by default;
Validate every hop after redirects;
Set DNS, connection, read, and total download timeouts;
Limit response body size and MIME type;
Disallow arbitrary local file paths;
Use an allowlist for trusted internal media domains.

These checks should happen at the media fetch layer, not when the model Worker receives malformed input.

A Practical Evolution Path

Step 1: Establish a Stage-Level Baseline First

Don’t change the architecture yet. Add stage latency, input scale, and resource metrics to your existing co-located deployment, covering at least one full real peak period. Key metrics include:

P50/P95/P99 for media_fetch_ms / media_decode_ms / vision_encode_ms;
Encoder queue depth;
Encoder GPU utilization / memory;
Image count, pixel volume, video frame count;
MM Processor Cache hit ratio;
Encoder Cache hit ratio;
TTFT distribution for text-only vs. multimodal requests.

You must separate text-only and multimodal requests. A blended overall P95 can easily mask the fact that vision requests have already degraded.

Step 2: Fix CPU / I/O First, Then Touch GPU Topology

If the bottleneck is concentrated in image fetch and decode, prioritize parallel media decoding, connection pooling, timeouts, media proxies, or edge caching. Adding Encoder GPUs at this point usually won’t solve the problem.

Step 3: Only Enable Encoder Cache When There’s Repeated Media

Measure the repeated-media ratio first, then decide on cache capacity. If 99% of requests carry unique images, a 10 GB embedding cache won’t deliver meaningful gains—it will just add host memory pressure and invalidation governance overhead.

Dynamo’s documentation states its multimodal embedding cache uses a CPU-side LRU; the current vLLM integration requires vLLM 0.17.0 or later. Always check the current version compatibility matrix for your deployment.

Step 4: Split Out the Encoder Only After It Saturates

If profiling shows the Encoder is a stable bottleneck, then split out Encode Workers. After splitting, re-validate:

TTFT = media + processor + queue(E) + encode + transfer + queue(LLM) + prefill

Don’t just check whether encode time dropped; confirm that the new queue(E) and transfer overhead haven’t eaten the gains.

Applicable Scenarios

This approach is especially suited for:

E-commerce product image Q&A, where the same product images are accessed repeatedly at scale;
OCR / image understanding platforms with multiple images per request;
Multi-turn Q&A over enterprise document screenshots, design mockups, or flowcharts;
Video understanding services that need strict sampling frame control;
Clusters serving both text-only and vision requests, where you want to prevent vision spikes from slowing text traffic;
Scenarios where the Vision Encoder and language model have significantly different GPU model, memory, or compute requirements.

Scenarios that are less suited to complex splitting from day one include low-concurrency internal tools, minimal media input, no obvious vision-stage bottleneck, or businesses still in rapid validation. In these cases, keeping a single co-located instance is usually more controllable.

Common Pitfalls

Pitfall 1: High TTFT → immediately add LLM GPUs. High multimodal TTFT may be caused by image download, decode, or vision Encoder blocking. Without stage metrics, adding GPUs is just expensive guesswork.

Pitfall 2: Enable a large cache whenever images are involved. Cache benefits depend on media repetition rate. Unique-image traffic won’t automatically get faster with a bigger LRU.

Pitfall 3: Treating Encoder Cache as Prefix / KV Cache. They cache different objects with completely different lifecycles and correctness boundaries. When the Encoder or Processor is upgraded, old embeddings generally cannot be reused unconditionally.

Pitfall 4: Going straight to full E/P/D disaggregation for architectural novelty. Each additional stage adds a queue, transfer path, failure domain, and capacity parameter. Prove the Encoder is the bottleneck first, then split.

Pitfall 5: Only limiting upload file size. A compressed JPEG can be tiny but decode to a huge pixel matrix; video file size also doesn’t directly represent actual sampled frames and Encoder cost. Budgets must target post-decode compute scale.

Pitfall 6: Treating media URLs as plain strings. Once the server actively fetches a URL, it’s a network access capability. Protocol, domain/IP, redirect, size, and timeout controls are mandatory.

Production Launch Checklist

References

NVIDIA Dynamo — Multimodal Model Serving: https://docs.nvidia.com/dynamo/v1.4.1/multimodal/overview
NVIDIA Dynamo — Parallel Media Decoding: https://docs.nvidia.com/dynamo/multimodal/parallel-media-decoding
NVIDIA Dynamo — Embedding Cache: https://docs.nvidia.com/dynamo/user-guides/multimodal/embedding-cache
NVIDIA Dynamo — Encoder Disaggregation: https://docs.nvidia.com/dynamo/dev/multimodal/encoder-disaggregation
vLLM — Multimodal Inputs: https://docs.vllm.ai/en/latest/features/multimodal_inputs/
Singh et al. — Efficiently Serving Large Multimodal Models Using EPD Disaggregation: https://arxiv.org/abs/2501.05460