concept:rank-world-size
Rank World Size
Rank World Size in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Distributed & fault-tolerant training
Choose data, sharded, tensor, or pipeline parallelism
Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDDP under the hood
Use ddp under the hood to move the distributed training production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.