concept:all-reduce

All Reduce

All Reduce in the context of production machine-learning systems.

Lessons

Formal explanations, examples, simulations, and checkpoints.

Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.

Use ddp under the hood to move the distributed training production brief toward a defensible release.

Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.

Use distributed checkpoints and fault-tolerant restarts to move the distributed training production brief toward a defensible release.

Use hangs, stragglers, and scaling efficiency to move the distributed training production brief toward a defensible release.

Dependency-authoritative learning units.

Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.

Assignment: Fault-tolerant distributed trainer

Reported facts and course reconstructions.

Distributed training performance is the overlap of compute, communication, input, and synchronization.

Parallelism is a placement problem constrained by bytes, shapes, and topology.