LLM Distributed Inference Communication Fault Debugging: Locating Collective Communication Hangs with NCCL RAS, Flight Recorder, and Topology Baselines
When NCCL collective communication hangs in large-scale distributed LLM inference, you're often left with only a vague timeout log. This article explains how to use RAS, Flight Recorder, topology baselines, and anomalous rank detection to build observable, rollback-capable communication gating for production deployments.