Training
The repeatable path from validated data to a versioned model artifact.
Lessons
Formal explanations, examples, simulations, and checkpoints.
From notebook to reproducible run
A reproducible result is a result whose complete input identity can be replayed.
Engineering foundationsContracts before models
Contracts keep bad meaning from becoming good-looking features.
Engineering foundationsTraining code is product code
The shipped model is the reloaded artifact, not the object still living in the trainer.
Engineering foundationsContainers and configuration without surprises
A container is reproducible only when its image, inputs, configuration, and outputs all have explicit identities.
Engineering foundationsObservable, idempotent, failure-safe workflows
A safe workflow makes commitment atomic and repetition uneventful.
Engineering foundationsRead a research paper as an executable specification
Turn a research paper into a testable claim map, implementation contract, and prioritized reading plan before writing code.
Engineering foundationsReproduce, validate, and productionize a paper
Implement a paper from baseline to novel component, diagnose reproduction gaps, and adapt it for production without silently changing its claim.
Neural networks from scratchBeyond linear boundaries
Hidden nonlinearities earn their place by making the final decision simpler in a learned representation.
Neural networks from scratchForward pass, activations, losses, and initialization
A trustworthy forward pass makes shapes, activation semantics, initialization, and loss expectations explicit.
Neural networks from scratchBackpropagation without magic
Backprop becomes ordinary once every edge in the forward graph has a local derivative you can inspect.
Neural networks from scratchSGD, momentum, AdamW, and schedules
The trained state is parameters plus optimizer history, schedule position, data order, and randomness.
Neural networks from scratchRegularize, normalize, debug, and ship
Shipping a network means freezing its inference behavior, not merely saving its weights.
Neural networks from scratchActivation functions: geometry, gradients, and failure modes
Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.
Neural networks from scratchLoss functions by task, data, and deployment constraints
Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.
GPU-aware trainingWhat the GPU actually executes
GPU performance comes from feeding a few substantial, well-shaped kernels—not merely from placing tensors on a CUDA device.
GPU-aware trainingKeep it fed: loaders and async copies
A fast training step needs an equally deliberate supply chain from storage to pinned memory to the device.
GPU-aware trainingMixed precision without NaNs
Mixed precision is trustworthy only when dtype choices, loss scaling, skipped steps, and quality parity are all observable.
GPU-aware trainingActivations, accumulation, and checkpointing
Checkpointing trades compute for activation memory; accumulation trades update frequency for batch memory—measure and validate each trade separately.
GPU-aware trainingProfile from timeline to cost per sample
Profile a controlled workload, explain the timeline, and judge improvements by cost to an unchanged quality target.
Courses & assignments
Dependency-authoritative learning units.
Engineering foundations
A seeded, tested, packaged, versioned training workflow with a model card.
Assignment: Reproducible baseline packageCourse 09Neural networks from scratch
A small network built from forward pass through backpropagation and optimization.
Assignment: Network from first principlesCourse 12GPU-aware training
A profiled, mixed-precision, memory-budgeted training loop.
Assignment: Single-GPU optimization planCasebook
Reported facts and course reconstructions.
Michelangelo: one ML platform across the lifecycle
A useful ML platform is a connected lifecycle, not a pile of independent tools.
Spotify · Data & platformThe winding road through TFX and Kubeflow
Platform adoption is a product and socio-technical problem.
Netflix · Data & platformMetaflow: a human-centric path from notebook to production
The best abstraction preserves the user’s reasoning model while adding production guarantees.
Uber · Training & distributionHorovod: make distributed training feel small
Distributed training performance is the overlap of compute, communication, input, and synchronization.
OpenAI · Training & distributionScaling Kubernetes to 7,500 nodes
At scale, metadata and coordination become a workload of their own.
NVIDIA · Training & distributionMegatron: composing parallelism for giant models
Parallelism is a placement problem constrained by bytes, shapes, and topology.
PyTorch · Training & distributionPerformant distributed checkpointing
A checkpoint is useful only when it is committed, portable, and routinely restored.
PyTorch · Training & distributionCompiling production training models safely
Compiler wins are workload-specific artifacts that need the same release discipline as models.
Google · Fairness, privacy & securityPrivate training for on-device language models
Privacy is an end-to-end guarantee with explicit adversary, accounting, and utility tradeoffs.