Operators, product owners, and people affected by the distributed training decision.
Training systems · weeks 16
Distributed & fault-tolerant training
An eight-GPU job is slower than one GPU and loses progress on failure.
Opening production brief
A measurable problem, before a model.
rank/world size, distributed samplers, all-reduce, global batch with version, owner, event time, and availability contracts.
An eight-GPU job is slower than one GPU and loses progress on failure. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
Choose data, sharded, tensor, or pipeline parallelism
Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.
65–90 min · checkpoint · deterministic lab - 02↗
DDP under the hood
DDP is replicated local training held together by a strict collective contract: same parameters, matched backward work, partitioned data, averaged gradients.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
FSDP and ZeRO-style sharding
Sharding reduces steady replicated state, but wrap boundaries decide the temporary memory and communication you pay to reconstruct it.
65–90 min · checkpoint · deterministic lab - 04↗
Distributed checkpoints and fault-tolerant restarts
Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Hangs, stragglers, and scaling efficiency
Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count.
65–90 min · checkpoint · deterministic lab - HW↗
Fault-tolerant distributed trainer
Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.
Rubric · staged hints · reference resolution
Prerequisite guidance
1 concepts make this faster.
Prerequisites are guidance, never hard gates.