concept:global-batch
Global Batch
Global Batch in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Distributed & fault-tolerant training
FSDP and ZeRO-style sharding
Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDistributed checkpoints and fault-tolerant restarts
Use distributed checkpoints and fault-tolerant restarts to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingHangs, stragglers, and scaling efficiency
Use hangs, stragglers, and scaling efficiency to move the distributed training production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.