concept:distributed-samplers
Distributed Samplers
Distributed Samplers in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Distributed & fault-tolerant training
Choose data, sharded, tensor, or pipeline parallelism
Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDDP under the hood
Use ddp under the hood to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingFSDP and ZeRO-style sharding
Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.