LLM Gradient Accumulation in Production: Token-Normalized Loss, DDP no_sync, and Scheduler Alignment to Avoid Equivalent Batch Distortion
Gradient accumulation seems like simple batch summing, but variable-length samples, distributed sync, mixed precision, and learning rate schedulers silently alter the equivalent batch. This article dives into production-grade solutions like token normalization, DDP no_sync, AMP update boundaries, and tail window handling to achieve mathematically consistent gradient accumulation in LLM fine-tuning.