Tag: Distributed Training

2 articles

• LLM Activation Checkpointing in Production: Selective Recompute, Non-Reentrant Mode, and RNG Consistency to Reduce Training Memory

  During large model training, activations often exhaust GPU memory before parameters. This article explores selective recomputation, non-reentrant mode, RNG state consistency, and engineering gating strategies to reduce peak memory while preserving gradient correctness and controlling throughput loss.

  07/14/2026 LLM Training / Activation Checkpointing / PyTorch / GPU Memory / Distributed Training

• LLM FP8 Fine-Tuning in Production: Amax Scaling, Layer Whitelists, and BF16 Fallback for Stable Convergence

  FP8 reduces compute and memory for LLM fine-tuning, but poor scaling, outliers, and distributed sync issues can cause silent accuracy degradation. This guide provides a complete engineering solution with Amax monitoring, layer whitelists, BF16 fallback, dual-track evaluation, and release gates.

  07/14/2026 LLM Training / FP8 / Transformer Engine / Fine-tuning / MLOps / Mixed Precision Training / Distributed Training / Amax Scaling