LLM Activation Checkpointing in Production: Selective Recompute, Non-Reentrant Mode, and RNG Consistency to Reduce Training Memory
During large model training, activations often exhaust GPU memory before parameters. This article explores selective recomputation, non-reentrant mode, RNG state consistency, and engineering gating strategies to reduce peak memory while preserving gradient correctness and controlling throughput loss.