All Reduce
All Reduce in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Choose data, sharded, tensor, or pipeline parallelism
Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.
Distributed & fault-tolerant trainingDDP under the hood
DDP is replicated local training held together by a strict collective contract: same parameters, matched backward work, partitioned data, averaged gradients.
Distributed & fault-tolerant trainingFSDP and ZeRO-style sharding
Sharding reduces steady replicated state, but wrap boundaries decide the temporary memory and communication you pay to reconstruct it.
Distributed & fault-tolerant trainingDistributed checkpoints and fault-tolerant restarts
Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery.
Distributed & fault-tolerant trainingHangs, stragglers, and scaling efficiency
Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.
Horovod: make distributed training feel small
Distributed training performance is the overlap of compute, communication, input, and synchronization.
NVIDIA · Training & distributionMegatron: composing parallelism for giant models
Parallelism is a placement problem constrained by bytes, shapes, and topology.