LLM GPU Self-Healing in Production: Reducing Inference Interruptions with Health Probes, Node Isolation, and Request Draining

Background: The “Zombie” GPU Is the Worst Nightmare for LLM Inference

In typical web services, restarting a crashed process usually fixes the issue. But LLM inference is fundamentally different—a single request can consume significant GPU memory, and the inference process holds model weights, KV cache, CUDA context, NCCL communication state, and a runtime scheduling queue.

When a GPU fails, the service doesn’t always crash immediately. The symptoms are often more subtle:

• Requests suddenly slow down, with high time-to-first-token (TTFT)
• Streaming output is interrupted
• GPU memory on a specific card is not released
• The container is alive, but CUDA calls fail
• Monitoring metrics are intermittently missing

If you rely solely on “process alive” checks, you’ll likely misdiagnose these issues—the Pod is still Running, /healthz returns 200, but new requests can’t execute reliably. Worse, after a simple restart triggered by a CUDA OOM, the node repeatedly joins the service and fails again, amplifying queueing and retry storms.

The goal of production-grade GPU self-healing isn’t “auto-restart everything,” but to establish a clear closed loop:

Detect anomaly → Stop accepting new requests → Drain in-flight requests → Isolate suspicious node → Rebuild inference process → Verify recovery → Preserve failure evidence

If any link in this chain is missing, a single-card problem can escalate into a cluster-wide disruption.

---

Core Principles: Decompose Health into Four Layers

Layer 1: Application Health

Application health answers: “Can the inference service process still accept and handle requests normally?” Typical signals include:

• Is the HTTP health endpoint reachable?
• Is the model fully loaded?
• Are the tokenizer, scheduler, and workers initialized?
• Is the queue in overload protection?
• Are request error rates, timeout rates, and cancellation rates abnormal over a recent window?

This layer is well-suited for Kubernetes readiness probes. The readiness probe’s purpose isn’t to restart the container, but to decide if the Pod should continue receiving Service traffic. For LLM Serving, the readiness semantics should be stricter: return not-ready if the model isn’t loaded, GPU workers are unavailable, the queue is at a protective watermark, or the node is draining.

Layer 2: GPU Device Health

GPU device health answers: “Can this GPU card be trusted to perform computations reliably?” Don’t just look at utilization and memory usage. NVIDIA DCGM health checks cover these dimensions:

Dimension	Check Content
PCIe	Bandwidth, replay count, link status
Memory	ECC errors, pending page retirement, faulty memory
Thermal	Temperature, throttling events
Power	Power anomalies, insufficient power supply
NVLink	Link errors, bandwidth degradation
Driver	Driver version, device visibility

DCGM documentation lists the following as failure conditions: GPU fallen off bus, severe XID, requires reset, requires reboot, pending page retirement, faulty memory. The application layer should handle these signals with different severity levels:

• warning → Reduce weight, observe, trigger migration
• error → Drain, isolate
• reset / reboot signals → Not just restart the container, but remove the node from the scheduling pool

Layer 3: Kubernetes Scheduling Health

Scheduling health answers: “Should this node still receive new GPU Pods?” The NVIDIA Kubernetes device plugin can expose GPU count and track health status, but the official documentation clearly states that the device plugin itself should not be considered a complete GPU fault self-healing system.

This means production environments typically need an additional GPU node controller. It reads DCGM, device plugin, Prometheus, Kubernetes events, and inference service metrics, and performs the following actions on nodes when necessary:

• cordon — Mark the node as unschedulable
• taint — Add an eviction taint
• Remove endpoints
• Trigger Pod rebuild or notify for manual intervention

Layer 4: Request Lifecycle Health

Request lifecycle health answers: “How do we handle in-flight requests when a failure occurs?” LLM requests differ from standard REST calls; they might be streaming output, have consumed many tokens, or be stuck in a tool call or long-context decode phase.

The self-healing logic must distinguish three types of requests:

1. Completed requests: Return normally, just record the node as suspicious.
2. In-flight requests: Prioritize completion or clean interruption; don’t leave connections hanging indefinitely.
3. Queued requests (not yet started): Migrate as soon as possible or return a retryable error.

Therefore, the self-healing process must include a drain mode. Once an instance enters drain mode, it stops accepting new requests and only handles existing ones. After exceeding the maximum drain time, it is forcefully terminated.

---

Engineering Implementation: An Executable GPU Self-Healing Pipeline

1. Define a Health State Machine

Don’t just use healthy=true/false. Break the instance state into these stages:

STARTING → WARMING → READY → DEGRADED → DRAINING → QUARANTINED → RECOVERING → READY
                                                                    ↘ FAILED

State	Meaning	Action
STARTING	Container starting, model not loaded	Wait for startup probe
WARMING	Model loaded, warming up	Probe inference, no production traffic
READY	Can accept production traffic	readiness=true, normal scheduling
DEGRADED	Warning signals detected	Reduce weight, limit new long requests, alert and observe
DRAINING	Stop accepting new requests, draining	readiness=false, load balancer removes traffic
QUARANTINED	Node or instance isolated	cordon/taint, excluded from scheduling
RECOVERING	Restarting worker or rebuilding Pod, verifying	Probe inference + canary recovery
FAILED	Auto-recovery failed	Manual intervention, preserve failure evidence

This state machine should drive the readiness probe, load balancing weights, alert levels, and operational actions.

2. Separate Readiness and Liveness

Many teams point both readiness and liveness probes to the same /healthz endpoint, which is a common mistake. LLM inference services should provide at least three endpoints:

Endpoint	Semantics	Failure Effect
/livez	Is the process alive?	Restart container
/readyz	Can it accept new requests?	Remove traffic
/gpuz	Are GPU workers, CUDA context, DCGM signals, and memory state trustworthy?	Trigger DRAINING

Example Kubernetes configuration:

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 2

livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  initialDelaySeconds: 120
  periodSeconds: 10
  failureThreshold: 3

startupProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 10
  failureThreshold: 60

Key point: Model loading and warmup can be slow. Use a startupProbe to prevent kubelet from killing the container before the model is loaded. Use readiness for traffic removal; don’t wait for the process to crash.

3. Trigger Node Isolation with DCGM Signals

Handle DCGM health results in three categories:

groups:
  - name: llm-gpu-health
    rules:
      - alert: LlmGpuMetricsMissing
        expr: absent_over_time(DCGM_FI_DEV_GPU_UTIL[3m])
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "GPU metrics disappeared for a serving node"

      - alert: LlmGpuXidError
        expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "GPU reported XID errors; drain and quarantine the node"

The key isn’t to copy these rules verbatim, but to grasp two concepts: monitor explicit hardware errors (XID, ECC, NVLink) and monitor structural anomalies—such as metrics suddenly disappearing, scrape delays, or changes in the number of devices.

4. Make Request Draining a First-Class Capability

The most common mistake in GPU fault self-healing is restarting directly, which causes three types of side effects:

• Streaming requests that have started outputting are abruptly disconnected.
• Client retries send traffic to other instances, creating a secondary peak.
• The failure scene is cleaned up, making root cause analysis impossible.

A safer draining process:

Controller marks instance as DRAINING
    → readiness probe fails, load balancer removes new traffic
    → Gateway stops assigning new requests to the instance
    → Instance continues processing requests already in decode
    → Queued but unexecuted requests are migrated or return a retryable error
    → Force terminate after exceeding max_drain_seconds
    → Save last error, GPU metric window, request samples, and container events
    → Rebuild worker or isolate node

Pod configuration direction:

terminationGracePeriodSeconds: 180
lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - "curl -X POST localhost:8080/admin/drain && sleep 30"

terminationGracePeriodSeconds must be greater than the maximum acceptable drain time; otherwise, kubelet will forcefully kill the process before requests are finished. In production, the Gateway and Controller must also be aware that the instance has entered drain mode, rather than just relying on a sleep inside the container.

5. Distinguish the Root Cause of CUDA OOM

CUDA OOM doesn’t mean the GPU is broken. At a minimum, distinguish four scenarios:

OOM Type	Typical Trigger	Recommended Action
Single request exceeds limits	Prompt or max_tokens too large	Reject request or reduce max_tokens
Batch overload	Scheduler puts too many tokens in at once	Reduce concurrency, reduce batch token limit
Memory fragmentation	Non-contiguous available memory after long runtime	Drain and rebuild worker
Driver/device anomaly	OOM accompanied by XID, CUDA context errors	Isolate node and perform GPU health check

If you let the liveness probe restart everything for every OOM, you’ll mask the real problem. The correct approach is to record the context of each OOM—input tokens, output tokens, batch size, KV cache usage, GPU ID, model version, adapter version, and recent DCGM signals—and then classify and handle it.

6. Design Node Isolation and Recovery Strategies

Node isolation shouldn’t rely solely on deleting Pods. Consider three layers:

Layer	Scope	Use Case
Instance isolation	Single inference process enters DRAINING	Single request anomaly, queue overload
Pod isolation	Pod is rebuilt, node can still schedule other GPU Pods	Memory fragmentation, process leak, worker deadlock
Node isolation	cordon + taint, no new GPU workloads allowed	Fallen off bus, severe XID, NVLink/driver-level failure

Don’t immediately restore full traffic after recovery. Use canary recovery: first, run probe inference → accept a small number of short requests → observe error rate, TTFT, TPOT, memory release, and DCGM signals → restore full weight after a stable window.

---

Applicable Scenarios

This approach is suitable for:

• Self-hosted vLLM, TensorRT-LLM, TGI, SGLang, or Ray Serve LLM inference clusters
• Multi-replica GPU inference services running on Kubernetes
• Scenarios with long outputs, streaming output, multi-tenant requests, or high-concurrency queues
• Clusters with a small number of GPU nodes, where a single node failure significantly impacts availability
• Engineering governance for inference interruptions, timeouts, retry storms, and node cascading failures

For development/test environments or single-node offline inference, you can simplify to: health checks, OOM logs, manual restarts, and basic alerts. But if your service faces users, you should at least implement readiness, drain, and failure evidence preservation.

---

Common Misconceptions

Misconception 1: Only Look at GPU Utilization

Normal GPU utilization doesn’t mean the node is healthy. Driver errors, device loss, metric gaps, PCIe/NVLink errors, and non-releasable memory can all occur before or independently of utilization changes. Production systems need to monitor numeric metrics, structural metrics, and request outcomes simultaneously.

Misconception 2: Auto-Restart Everything

Auto-restart is suitable for process-level failures, not all GPU failures. For GPU problems that require a reset or reboot, a container restart is usually insufficient; if you continue scheduling new Pods to that node, they will just fail repeatedly.

Misconception 3: Draining Only via Kubernetes Pod Deletion

Kubernetes’ graceful termination provides a basic termination flow, but it doesn’t know if an LLM request is still decoding, streaming output, or can be migrated. Draining must be coordinated by the Gateway, Serving Runtime, and Controller.

Misconception 4: Full Traffic Immediately After Recovery

After GPU fault recovery, you should first run a canary. Especially after a driver reset, node restart, or model reload, you should first verify with probe inference, short requests, long requests, and concurrent requests, rather than restoring all traffic immediately.

---

Go-Live Checklist

Health Probes

• Are /livez, /readyz, and /gpuz semantically separated?
• Does startupProbe cover model loading and warmup time?
• Does readinessProbe fail during drain, overload, or GPU worker anomalies?
• Does livenessProbe avoid killing instances that are in a long warmup phase?

GPU Monitoring

• Are DCGM or equivalent GPU health metrics integrated?
• Are XID, fallen off bus, reset required, and pending page retirement monitored?
• Are metric gaps, scrape delays, and changes in GPU device count monitored?
• Are warning, error, and fatal signals mapped to different actions?

Request Draining

• Does the Gateway support instance-level traffic removal?
• Can queued requests be migrated or fast-failed?
• Can streaming requests be cleanly ended or return an explainable error?
• Is terminationGracePeriodSeconds greater than the maximum drain window?

Node Isolation

• Is cordon/taint supported for suspicious GPU nodes?
• Is there a distinction between Pod rebuild, GPU reset, node reboot, and manual repair?
• Is there a probe inference before recovery and a stable window after recovery?
• Are the last segment of metrics, logs, Kubernetes events, and request samples preserved?

---

References

• NVIDIA DCGM Diagnostics
• NVIDIA DCGM Feature Overview / Health Monitor
• NVIDIA Kubernetes Device Plugin
• Kubernetes Pod Lifecycle
• Ray Serve Production Guide
• When GPUs Fail Quietly: Observability-Aware Early Warning Beyond Numeric Telemetry