concept:global-batch
Global Batch
Global Batch in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Distributed & fault-tolerant training
FSDP and ZeRO-style sharding
Sharding reduces steady replicated state, but wrap boundaries decide the temporary memory and communication you pay to reconstruct it.
Distributed & fault-tolerant trainingDistributed checkpoints and fault-tolerant restarts
Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery.
Distributed & fault-tolerant trainingHangs, stragglers, and scaling efficiency
Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.