LLM GPU Multi-Tenant Isolation in Production: Sharing Inference Stably with MIG Profiles, Time-Slicing Layering, and Reconfiguration Gates

Background: Sharing a GPU Doesn’t Mean Sharing It Safely

When deploying LLM inference services in production, enterprises often face two pressures simultaneously: on one hand, large GPUs are costly, and a single low-traffic model occupying an entire card for long periods wastes utilization; on the other hand, simply cramming multiple tenants or models onto the same GPU can lead to memory contention, tail latency jitter, and fault propagation.

The key issue isn’t “can multiple processes see the same GPU,” but rather understanding three distinct sharing semantics:

• MIG (Multi-Instance GPU): Partitions supported NVIDIA GPUs into multiple predefined hardware instances, each with dedicated compute and memory resources.
• Time-Slicing: Allows multiple processes or Pods to share the same GPU by time-multiplexing, increasing concurrent access but providing no memory or fault isolation.
• MPS (Multi-Process Service): Reduces process switching by sharing the CUDA scheduling context and allows kernel and copy operations from multiple CUDA processes to overlap.

These are not three interchangeable “performance switches.” Production design must first determine the isolation level, then decide how to optimize resource utilization.

Core Principles: Define the Isolation Boundary First, Then Talk About Overcommitment

MIG: Carving a Physical GPU into Schedulable Hardware Instances

NVIDIA MIG partitions supported GPUs into isolated instances, each with independent compute and memory resources, exposed to CUDA and Kubernetes as if they were separate GPUs. For online inference requiring clear memory caps, stable latency, and tenant fault boundaries, MIG is typically the better foundational layer.

The cost of MIG is fixed resource granularity. Available profiles are determined by the specific GPU model; you cannot arbitrarily specify any number of cores or memory capacity like CPU requests. Therefore, the engineering focus shifts from “assigning a GPU to a Pod” to “selecting the right MIG profile for the model.”

Time-Slicing: Increasing Access Density Without Creating Isolation

Time-Slicing advertises a single GPU as multiple allocatable replicas, allowing multiple Pods to run in turns. NVIDIA’s official documentation explicitly states that these replicas have no memory isolation and share the same fault domain; requesting multiple shared replicas does not guarantee proportional compute power.

Therefore, Time-Slicing is better suited for:

• Trustworthy workloads from the same team;
• Low-occupancy, short-burst development or testing tasks;
• Non-critical inference where tail latency interference is acceptable;
• Older GPU models that don’t support MIG.

It should not be marketed as a “logical mini-GPU” and is not suitable for directly hosting untrusted external tenants.

MPS: Reducing Multi-Process Switching Overhead, But Not a Complete Tenant Isolation Solution

MPS coordinates multiple CUDA clients through a control daemon and shared service process, reducing context switching and allowing kernels and memory copies from different processes to overlap more easily. It’s suitable for multiple cooperative CUDA processes aiming to improve GPU utilization together.

However, in the Kubernetes NVIDIA Device Plugin, MPS sharing remains experimental and cannot currently be used simultaneously with MIG-enabled devices. Production environments should not stack all sharing mechanisms; choose one primary path based on isolation and stability requirements.

Profile Selection: Don’t Just Look at the Model Weight File Size

The memory requirements for an LLM inference instance include at least:

required_memory = model_weights + peak_kv_cache + runtime_workspace + allocator_and_safety_headroom

The most easily underestimated component is the peak KV Cache. The same model can have vastly different memory requirements for short Q&A versus long-context, high-concurrency scenarios. Profile selection should be based on target traffic replay, not just reading the safetensors file size.

It’s recommended to create a profile characterization for each model version:

Dimension	Description
Model precision and quantization format	FP16 / INT8 / GPTQ, etc.
Max input and output tokens	Directly affects KV Cache upper limit
Target concurrency	Determines peak memory for concurrent requests
Peak/steady-state memory and OOM boundary	Measured stress test data
TTFT, TPOT, and P95/P99 latency	SLO validation metrics
Allowed sharing mode	Exclusive MIG / Time-Slicing within MIG / Full GPU sharing

A profile is not a static configuration item; it’s a deployment artifact determined by the model version, context strategy, and concurrency targets.

Kubernetes Engineering Implementation

1. Managing MIG with GPU Operator

The GPU Operator manages MIG mode and geometry configuration on nodes via the MIG Manager. The cluster can choose a single or mixed strategy:

• single: All MIG GPUs on a node use a uniform profile;
• mixed: A node can expose multiple MIG resource types, suitable for hosting models of different sizes simultaneously.

Example installation:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set mig.strategy=mixed

Apply a balanced MIG configuration to a node:

kubectl label node <node-name> \
  nvidia.com/mig.config=all-balanced \
  --overwrite

Business Pods should explicitly request a specific profile, not vaguely request a full GPU:

apiVersion: v1
kind: Pod
metadata:
  name: llm-small-model
spec:
  containers:
  - name: server
    image: example/llm-server:stable
    resources:
      limits:
        nvidia.com/mig-2g.20gb: 1

The actual resource name depends on the GPU model and the profiles exposed by the mixed strategy; confirm via the node’s resource list before deployment.

2. Separate Shared Resource Names from Exclusive Ones

If Time-Slicing is enabled, it’s recommended to set renameByDefault: true to expose shared resources as .shared, preventing applications from mistakenly believing they have an exclusive GPU. Also enable failRequestsGreaterThanOne to prevent businesses from inferring compute quotas by requesting multiple replicas.

version: v1
sharing:
  timeSlicing:
    renameByDefault: true
    failRequestsGreaterThanOne: true
    resources:
    - name: nvidia.com/gpu
      replicas: 2

Resource naming itself should convey the isolation semantics:

Resource Name	Meaning
nvidia.com/gpu	Exclusive full GPU
nvidia.com/mig-2g.20gb	Exclusive MIG instance
nvidia.com/gpu.shared	Shared time-slice access

Don’t use the same resource name for all three levels and rely on verbal agreements to distinguish them.

3. Treat MIG Reconfiguration as a Disruptive Infrastructure Change

The GPU Operator documentation states that the MIG Manager terminates GPU Pods before changing the configuration; some cloud environments may also require a node reboot to switch MIG modes. Therefore, MIG reconfiguration must go through a complete gate:

1. Freeze new scheduling: cordon the target node;
2. Migrate traffic: confirm that model replicas on other nodes have taken over traffic;
3. Drain GPU workloads: avoid reconfiguration during active requests;
4. Apply new profile: update nvidia.com/mig.config;
5. Wait for success state: check nvidia.com/mig.config.state=success;
6. Validate resources: verify nvidia-smi -L, node Capacity, and Allocatable;
7. Smoke test and stress test: verify loading, memory, latency, and error rates;
8. Restore scheduling: uncordon and gradually reintroduce traffic.

When reconfiguration fails, the rollback target is not a single Pod but the entire node’s MIG geometry. You must retain the previous configuration and a node-level rollback script.

Scheduling Layering: Choose the Sharing Method Based on Business Risk

It’s recommended to split GPU node pools by isolation level:

Node Pool	Sharing Method	Suitable Workloads	Main Boundary
gpu-exclusive	Full GPU exclusive	Very large models, cross-GPU parallelism, critical low-latency services	Highest cost, clearest isolation
gpu-mig	MIG Profile	Multi-tenant online inference, stable service for small/medium models	Fixed profiles, reconfiguration causes disruption
gpu-mig-shared	MIG + Time-Slicing	Low-load models within the same trust domain	Hard isolation between instances, contention within instances
gpu-timeslice	Full GPU Time-Slicing	Development, testing, low-priority tasks	No memory or fault isolation

This is easier to audit and roll back than making ad-hoc decisions on the same node like “share today, exclusive tomorrow.”

Monitoring: Don’t Just Look at GPU Utilization

Monitoring shared inference needs to distinguish between three layers: the physical GPU, MIG instances, and business requests.

Infrastructure Layer

• MIG configuration status and profile count;
• GPU/MIG instance memory usage;
• SM utilization, power, temperature, and XID errors;
• Node Capacity, Allocatable, and actual Pod allocation;
• MIG reconfiguration duration and failure count.

Inference Service Layer

• Queue length per model replica;
• TTFT, TPOT, end-to-end P95/P99;
• Active sequence count, token throughput, and OOMs;
• Loading failures due to undersized profiles;
• Neighbor interference within the same shared instance.

A special note: The NVIDIA GPU Operator documentation states that in Device Plugin Time-Slicing mode, DCGM Exporter cannot associate metrics with specific containers. This means tenant-level attribution is weak in full-GPU sharing mode; critical businesses should not rely solely on this mode for fine-grained billing or SLO accountability.

Common Misconceptions

Misconception 1: 8 GPU Resources in Kubernetes Means 8x Compute Power

Time-Slicing replicas are just shared access credentials, not independent compute units. The resource count is inflated, but compute, memory, or bandwidth do not scale proportionally.

Misconception 2: MIG Solves All Performance Jitter

MIG provides stronger resource and fault isolation, but jitter can still occur within an application due to batching, KV Cache, CPU-GPU data paths, and model configuration. MIG is an isolation foundation, not a substitute for inference performance tuning.

Misconception 3: Smaller Profiles Always Mean Higher GPU Utilization

An overly small profile can prevent model loading, limit KV Cache capacity, or reduce concurrency. Compare effective token throughput per physical GPU, SLO-compliant request count, and idle fragmentation, not just the number of instances.

Misconception 4: MIG Geometry Can Be Changed Frequently Based on Real-Time Traffic

The official management process stops GPU workloads, and some environments require node reboots. MIG geometry is suitable for adjustment based on capacity cycles or maintenance windows, not as part of an online request scheduling loop.

Go-Live Checklist

• Confirmed GPU model supports MIG and verified available profiles;
• Recorded model weights, peak KV Cache, workspace, and safety headroom;
• Validated profile through target context and concurrency replay;
• Exclusive, MIG, and shared resources use different resource names and node pools;
• Time-Slicing has sharing identifiers enabled and prevents requesting multiple replicas to create a quota illusion;
• Reconfiguration process includes cordon, drain, validation, rollback, and maintenance window;
• Verified MIG status, node resources, and nvidia-smi -L are consistent;
• Established alerts for instance-level memory, latency, OOM, XID, and configuration status;
• Understood container-level monitoring and billing limitations under Time-Slicing;
• Prepared a rollback path from shared nodes to exclusive or larger MIG profiles.

Frequently Asked Questions

Is MPS better than Time-Slicing for LLM online inference?

Not necessarily. MPS can reduce context switching and limit some compute and memory resources for clients, but the Kubernetes Device Plugin currently marks MPS support as experimental and does not support it on MIG-enabled devices. Critical online services should first validate drivers, fault propagation, monitoring, and recovery paths.

How do I decide between MIG and full GPU exclusivity?

When a single model requires cross-GPU parallelism, uses most of the GPU memory, or is extremely sensitive to tail latency, full GPU exclusivity is usually simpler. When multiple small/medium models need clear memory and fault boundaries, prioritize MIG. Only use Time-Slicing in trustworthy, low-priority scenarios where mutual interference is acceptable.

References

1. NVIDIA Multi-Instance GPU User Guide
2. NVIDIA GPU Operator with MIG
3. NVIDIA GPU Operator: Time-Slicing GPUs in Kubernetes
4. NVIDIA Kubernetes Device Plugin
5. NVIDIA Multi-Process Service Documentation