Why can't VLM Serving simply reuse text LLM rate-limiting rules?

Images and videos introduce variables like visual encoding, resolution, frame count, and vision token budgets. The same number of requests can have vastly different GPU, CPU, network, and storage costs. QPS/RPM limits alone cannot accurately describe the true resource consumption of multimodal requests.

What dimensions should a vision token budget control?

At minimum, you should simultaneously control per-image resolution, number of images per request, video frame extraction count, detail level, total estimated visual tokens, and tenant-level budgets. Continuously calibrate these against real usage data.

What is most commonly overlooked in production VLM serving?

The most overlooked aspects are queuing during input preprocessing and visual encoding, handling anomalous images, protecting against oversized videos, cache hit rates, and attribution of vision token costs. These are often the root causes of tail latency and cost overruns.

VLM Serving in Production: Stabilizing Image & Video Inference with Vision Token Budgets

Background: VLM Inference Is Not Just “Passing an image_url”

More and more businesses are integrating Vision-Language Models (VLMs) into production systems: invoice recognition, screenshot Q&A, quality inspection, product image understanding, customer service ticket image analysis, video clip summarization, UI automation testing, and medical or industrial image-assisted analysis.

From an API perspective, multimodal calls might seem like simply changing the content in a text message from a plain string to “text + image URL” or “text + video URL.” But once in production, the risks of VLM Serving quickly surface:

One request with a single 512px image and another with 6 high-resolution screenshots have completely different costs.
Video input introduces additional variables like frame extraction, decoding, frame limits, upload size, and preprocessing queuing.
High-resolution images can significantly amplify the number of visual tokens, impacting latency, VRAM, and cost.
Text LLM QPS/RPM rate limiting is insufficient to describe the true resource consumption of multimodal requests.
Input preprocessing failures, abnormal image formats, slow external downloads, and oversized video files all degrade service tail latency.

Therefore, the core of VLM Serving isn’t “can the model be called,” but whether you can establish a production system based on Vision Token Budgets + Input Governance + Multi-Stage Queues + Cost Attribution.

Key Fact: Visual Input Ultimately Becomes Model Context Cost

The engineering path for most VLMs can be summarized as:

Receive text, image, or video input.
Download or read media files.
Decode, resize, crop, and normalize images.
Extract frames or construct frame sequences for videos.
Generate visual representations via a vision encoder or multimodal processor.
Concatenate or fuse the visual representation with the text prompt and feed it to the language model.
Output a text response.

The vLLM documentation on multimodal inputs explains that it supports passing images, videos, audio, embeddings, and cached inputs to multimodal models. It uses limit_mm_per_prompt or --limit-mm-per-prompt.image to control the number of multimodal inputs per prompt. The SGLang Vision API documentation also demonstrates the OpenAI-compatible vision input format and supports multiple images and interleaved text-image inputs. The OpenAI Images and Vision documentation clearly defines detail levels, image scaling, and image token cost calculation rules.

This means production systems must treat VLM input as a measurable, budgetable, and schedulable resource, not simply as a regular request parameter.

Core Design: Establish Vision Token Budgets at the Gateway Layer

The first layer of governance for VLM Serving should be at the entry point, not after the request reaches the GPU worker. It’s recommended to add a multimodal admission controller to your LLM Gateway or API Gateway.

It should, at minimum, perform the following checks:

Check Item	Goal	Handling Strategy
File Type	Block unexpected media input	Only allow whitelisted formats like jpg/png/webp/mp4
File Size	Prevent uploads/downloads from overwhelming the service	Reject or require compression if exceeded
Image Count	Prevent multi-image requests from stacking indefinitely	Configure limits per model and tenant
Image Resolution	Control vision token cost	Resize, downgrade detail, or reject if exceeded
Video Duration	Control frame extraction and preprocessing cost	Limit seconds, frames, or only use key segments
Detail Level	Balance quality, cost, and latency	Default to low/auto; require explicit request for high-precision scenarios
Tenant Budget	Enable cost attribution and quota governance	Deduct budget based on estimated visual token count

A practical budget model can start coarse-grained:

estimated_visual_cost = image_count × image_detail_factor × resolution_factor
                      + video_frame_count × frame_detail_factor
                      + preprocessing_cost_factor

This is not a final billing formula but an engineering estimate for admission control and queuing scheduling. Once in production, calibrate this estimate using actual usage returned by the model, GPU metrics, and preprocessing time.

Input Governance: Standardize First, Then Enter the Inference Queue

VLM services should generally not have GPU workers directly handle raw images or videos. A safer approach is to split the process into three stages.

Stage 1: Media Access and Security Checks

The entry layer is responsible for validating URLs, Content-Type, file size, media format, and tenant permissions. For external image URLs, set download timeouts, redirect limits, private network address blocking, maximum response body size, and MIME sniffing to prevent SSRF, slow downloads, and anomalous files from overwhelming the service.

Stage 2: Preprocessing and Visual Budget Estimation

The preprocessing layer handles image decoding, EXIF processing, transparent background handling, resizing, cropping, video frame extraction, anomalous frame filtering, and visual token estimation. This stage can use CPU workers or a dedicated preprocessing pool and should not be mixed with GPU inference workers.

Stage 3: Multimodal Inference Queue

Only structured input that has been governed should enter the inference queue. For example:

{
  "tenant_id": "enterprise-a",
  "model": "qwen2.5-vl-7b-instruct",
  "text_tokens_estimated": 860,
  "image_count": 2,
  "video_frame_count": 0,
  "detail": "low",
  "visual_tokens_estimated": 1200,
  "priority": "standard"
}

This allows the scheduler to make more intelligent queuing decisions based on text tokens, visual tokens, tenant priority, and GPU state, rather than just request count.

Why Separate Queues for Images, Videos, and Text Requests

In text LLM services, a request’s cost is primarily determined by prompt tokens, output tokens, and KV cache usage. VLM services add stages for media download, decoding, image processing, video frame extraction, and visual encoding.

If all requests enter the same queue, you may encounter these problems:

Large video requests block simple image Q&A.
Multi-image requests increase queuing latency, affecting lightweight screenshot Q&A.
Failed or slow image downloads occupy the inference entry point.
The vision encoder becomes a bottleneck while the GPU language model remains idle.
Cost attribution only sees “request count,” not the resource differences caused by visual input.

A better strategy is to use four types of queues:

download queue: Handles external link downloads and uploaded file storage.
preprocess queue: Handles decoding, resizing, frame extraction, and budget estimation.
vision encoder queue: Handles visual encoding or image embedding.
llm inference queue: Handles the final text generation.

Metrics for these four queue types should be observed separately. Pay special attention to:

Metric	Description
preprocess p95	P95 latency of the preprocessing stage
vision encoder queue length	Queue length for the vision encoder
visual token estimate	Deviation between estimated and actual visual tokens
media fetch failure rate	Rate of media download failures
video frame count distribution	Distribution of extracted video frames
detail level distribution	Proportion of requests at each detail level

Detail Level: Don’t Default All Images to Highest Precision

The OpenAI Images and Vision documentation categorizes detail into levels like low, high, original, and auto, explaining that low is suitable for fast, low-cost understanding that doesn’t require fine-grained visual details. Some models use different scaling, patch, or tiling calculations based on the detail level and image size.

This has direct implications for self-built VLM Serving:

Screenshot summary, image classification, object presence detection: Default to low or automatic downsampling.
OCR, tables, invoices, contract screenshots: Allow high, but limit resolution and number of images.
UI click targeting, dense chart understanding, industrial quality inspection details: Allow higher detail, but require a higher budget and stricter rate limiting.
Video summarization: Implement a frame extraction strategy; don’t feed all frames to the model.

In production, the detail level should not be a parameter the client can freely set. It should be determined server-side based on the business scenario, tenant tier, budget, and model capabilities.

Video Input: The Real Cost Is Usually in the Frame Extraction Strategy

Video VLM services are more prone to cost overruns than image services. A 30-second video, if extracted at 1 frame per second, yields 30 frames; at 5 frames per second, it yields 150 frames. Even if the model supports video input, engineering must clarify:

Maximum video duration.
Maximum number of extracted frames.
Frame extraction interval or keyframe strategy.
Whether multiple video inputs are allowed.
Maximum video resolution.
Whether to process long videos asynchronously.
Whether to convert long video tasks into batch processing tasks.

The vLLM documentation provides usage examples for video input, but “supporting video input” does not mean “production can accept unlimited video input.” When going live, treat video requests as high-cost tasks with separate governance.

Engineering Architecture for Implementation

A recommended architecture is:

Client → LLM Gateway → Media Admission Controller → Media Fetcher / Uploader
       → Image & Video Preprocessor → Visual Token Estimator
       → Multimodal Scheduler → VLM Worker Pool → Usage Recorder / Cost Attribution

Each layer’s responsibilities should be clear:

Media Admission Controller: Checks format, size, count, resolution, detail, and budget.
Media Fetcher: Downloads external URLs with timeouts, size limits, and security isolation.
Preprocessor: Standardizes image orientation, color, scaling, video frame extraction, and handles anomalies.
Visual Token Estimator: Estimates visual input cost before inference.
Multimodal Scheduler: Schedules based on visual tokens, text tokens, tenant priority, and queue state.
VLM Worker Pool: Executes the actual visual encoding and text generation.
Usage Recorder: Records actual tokens, latency, failure reasons, and tenant costs.

Common Pitfalls

Pitfall 1: Rate Limiting Only by QPS

The cost variance of VLM requests is far greater than that of regular text requests. 1 QPS of high-resolution multi-image OCR can be heavier than 20 QPS of short text Q&A. Rate limiting must at least incorporate visual token estimation, image count, video frame count, and detail level weighting.

Pitfall 2: Placing Image Preprocessing Inside GPU Workers

This forces GPU workers to handle downloading, decoding, resizing, anomaly handling, and inference logic, leading to unclear failure boundaries. Preprocessing should be front-loaded and horizontally scalable.

Pitfall 3: Using a Single Default Detail Level for All Business Scenarios

Different businesses have vastly different needs for visual detail. Customer service image classification and invoice OCR should not use the same detail strategy.

Pitfall 4: Ignoring the Cost of Failed Inputs

Corrupted images, slow URLs, oversized images, damaged videos, and incorrect MIME types all consume system resources. Failed requests must also be logged for cost and frequency; otherwise, troubleshooting will only reveal “the model is slow.”

Go-Live Checklist

Before going live, at least check the following items:

Conclusion

The production challenge of VLM Serving is not just “whether the model supports images or videos,” but how to make images, videos, and text inputs all governable resources.

A stable multimodal inference service should complete media security checks at the entry layer, standardization and visual token estimation at the preprocessing layer, differentiation of heavy and light requests at the scheduling layer, and recording of true costs and failure reasons at the observability layer. Only then can a team stably control latency, VRAM, cost, and service quality under the mixed load of multiple images, high-resolution screenshots, video frame extraction, and different tenants.

Key References

vLLM Multimodal Inputs
SGLang OpenAI APIs - Vision
OpenAI Images and Vision Guide
LLaVA-OneVision: arXiv:2408.03326
LLaVA-Mini: arXiv:2501.03895
Flamingo: a Visual Language Model for Few-Shot Learning: arXiv:2204.14198