tool:pytorch

PyTorch

A tensor, autograd, neural-network, and distributed-training runtime.

Lessons

Formal explanations, examples, simulations, and checkpoints.

Neural networks from scratch

Beyond linear boundaries

Hidden nonlinearities earn their place by making the final decision simpler in a learned representation.

Neural networks from scratch

Forward pass, activations, losses, and initialization

A trustworthy forward pass makes shapes, activation semantics, initialization, and loss expectations explicit.

Neural networks from scratch

Backpropagation without magic

Backprop becomes ordinary once every edge in the forward graph has a local derivative you can inspect.

Neural networks from scratch

SGD, momentum, AdamW, and schedules

The trained state is parameters plus optimizer history, schedule position, data order, and randomness.

Neural networks from scratch

Regularize, normalize, debug, and ship

Shipping a network means freezing its inference behavior, not merely saving its weights.

Neural networks from scratch

Activation functions: geometry, gradients, and failure modes

Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.

Neural networks from scratch

Loss functions by task, data, and deployment constraints

Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.

Distributed & fault-tolerant training

Choose data, sharded, tensor, or pipeline parallelism

Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.

Distributed & fault-tolerant training

DDP under the hood

DDP is replicated local training held together by a strict collective contract: same parameters, matched backward work, partitioned data, averaged gradients.

Distributed & fault-tolerant training

FSDP and ZeRO-style sharding

Sharding reduces steady replicated state, but wrap boundaries decide the temporary memory and communication you pay to reconstruct it.

Distributed & fault-tolerant training

Distributed checkpoints and fault-tolerant restarts

Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery.

Distributed & fault-tolerant training

Hangs, stragglers, and scaling efficiency

Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count.