Operators, product owners, and people affected by the distributed training decision.
Training systems · weeks 16
Distributed & fault-tolerant training
An eight-GPU job is slower than one GPU and loses progress on failure.
Opening production brief
A measurable problem, before a model.
rank/world size, distributed samplers, all-reduce, global batch with version, owner, event time, and availability contracts.
An eight-GPU job is slower than one GPU and loses progress on failure. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
Choose data, sharded, tensor, or pipeline parallelism
Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 02↗
DDP under the hood
Use ddp under the hood to move the distributed training production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
FSDP and ZeRO-style sharding
Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 04↗
Distributed checkpoints and fault-tolerant restarts
Use distributed checkpoints and fault-tolerant restarts to move the distributed training production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Hangs, stragglers, and scaling efficiency
Use hangs, stragglers, and scaling efficiency to move the distributed training production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - HW↗
Fault-tolerant distributed trainer
Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.
Rubric · staged hints · reference resolution
Prerequisite guidance
1 concepts make this faster.
Prerequisites are guidance, never hard gates.