concept:distributed-samplers

Distributed Samplers

Distributed Samplers in the context of production machine-learning systems.

Lessons

Formal explanations, examples, simulations, and checkpoints.

Choose data, sharded, tensor, or pipeline parallelism

Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.

Distributed & fault-tolerant training

DDP under the hood

DDP is replicated local training held together by a strict collective contract: same parameters, matched backward work, partitioned data, averaged gradients.

Distributed & fault-tolerant training

FSDP and ZeRO-style sharding

Sharding reduces steady replicated state, but wrap boundaries decide the temporary memory and communication you pay to reconstruct it.

Courses & assignments

Dependency-authoritative learning units.

Casebook

Reported facts and course reconstructions.