concept:rank-world-size
Rank World Size
Rank World Size in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Distributed & fault-tolerant training
Choose data, sharded, tensor, or pipeline parallelism
Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.
Distributed & fault-tolerant trainingDDP under the hood
DDP is replicated local training held together by a strict collective contract: same parameters, matched backward work, partitioned data, averaged gradients.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.