PyTorch
A tensor, autograd, neural-network, and distributed-training runtime.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Beyond linear boundaries
Use beyond linear boundaries to move the neural networks production brief toward a defensible release.
Neural networks from scratchForward pass, activations, losses, and initialization
Use forward pass, activations, losses, and initialization to move the neural networks production brief toward a defensible release.
Neural networks from scratchBackpropagation without magic
Use backpropagation without magic to move the neural networks production brief toward a defensible release.
Neural networks from scratchSGD, momentum, AdamW, and schedules
Use sgd, momentum, adamw, and schedules to move the neural networks production brief toward a defensible release.
Neural networks from scratchRegularize, normalize, debug, and ship
Use regularize, normalize, debug, and ship to move the neural networks production brief toward a defensible release.
Neural networks from scratchActivation functions: geometry, gradients, and failure modes
Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.
Neural networks from scratchLoss functions by task, data, and deployment constraints
Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.
Distributed & fault-tolerant trainingChoose data, sharded, tensor, or pipeline parallelism
Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDDP under the hood
Use ddp under the hood to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingFSDP and ZeRO-style sharding
Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDistributed checkpoints and fault-tolerant restarts
Use distributed checkpoints and fault-tolerant restarts to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingHangs, stragglers, and scaling efficiency
Use hangs, stragglers, and scaling efficiency to move the distributed training production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Neural networks from scratch
A small network built from forward pass through backpropagation and optimization.
Assignment: Network from first principlesCourse 13Distributed & fault-tolerant training
Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.
Assignment: Fault-tolerant distributed trainerCasebook
Reported facts and course reconstructions.
Horovod: make distributed training feel small
Distributed training performance is the overlap of compute, communication, input, and synchronization.
PyTorch · Training & distributionPerformant distributed checkpointing
A checkpoint is useful only when it is committed, portable, and routinely restored.
PyTorch · Training & distributionCompiling production training models safely
Compiler wins are workload-specific artifacts that need the same release discipline as models.