Training
The repeatable path from validated data to a versioned model artifact.
Lessons
Formal explanations, examples, simulations, and checkpoints.
From notebook to reproducible run
Use from notebook to reproducible run to move the engineering foundations production brief toward a defensible release.
Engineering foundationsContracts before models
Use contracts before models to move the engineering foundations production brief toward a defensible release.
Engineering foundationsTraining code is product code
Use training code is product code to move the engineering foundations production brief toward a defensible release.
Engineering foundationsContainers and configuration without surprises
Use containers and configuration without surprises to move the engineering foundations production brief toward a defensible release.
Engineering foundationsObservable, idempotent, failure-safe workflows
Use observable, idempotent, failure-safe workflows to move the engineering foundations production brief toward a defensible release.
Engineering foundationsRead a research paper as an executable specification
Turn a research paper into a testable claim map, implementation contract, and prioritized reading plan before writing code.
Engineering foundationsReproduce, validate, and productionize a paper
Implement a paper from baseline to novel component, diagnose reproduction gaps, and adapt it for production without silently changing its claim.
Neural networks from scratchBeyond linear boundaries
Use beyond linear boundaries to move the neural networks production brief toward a defensible release.
Neural networks from scratchForward pass, activations, losses, and initialization
Use forward pass, activations, losses, and initialization to move the neural networks production brief toward a defensible release.
Neural networks from scratchBackpropagation without magic
Use backpropagation without magic to move the neural networks production brief toward a defensible release.
Neural networks from scratchSGD, momentum, AdamW, and schedules
Use sgd, momentum, adamw, and schedules to move the neural networks production brief toward a defensible release.
Neural networks from scratchRegularize, normalize, debug, and ship
Use regularize, normalize, debug, and ship to move the neural networks production brief toward a defensible release.
Neural networks from scratchActivation functions: geometry, gradients, and failure modes
Select activation functions by signal range, gradient flow, architecture, precision, and failure mode; then diagnose saturation and dead units from telemetry.
Neural networks from scratchLoss functions by task, data, and deployment constraints
Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.
GPU-aware trainingWhat the GPU actually executes
Use what the gpu actually executes to move the gpu training production brief toward a defensible release.
GPU-aware trainingKeep it fed: loaders and async copies
Use keep it fed: loaders and async copies to move the gpu training production brief toward a defensible release.
GPU-aware trainingMixed precision without NaNs
Use mixed precision without nans to move the gpu training production brief toward a defensible release.
GPU-aware trainingActivations, accumulation, and checkpointing
Use activations, accumulation, and checkpointing to move the gpu training production brief toward a defensible release.
GPU-aware trainingProfile from timeline to cost per sample
Use profile from timeline to cost per sample to move the gpu training production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Engineering foundations
A seeded, tested, packaged, versioned training workflow with a model card.
Assignment: Reproducible baseline packageCourse 09Neural networks from scratch
A small network built from forward pass through backpropagation and optimization.
Assignment: Network from first principlesCourse 12GPU-aware training
A profiled, mixed-precision, memory-budgeted training loop.
Assignment: Single-GPU optimization planCasebook
Reported facts and course reconstructions.
Michelangelo: one ML platform across the lifecycle
A useful ML platform is a connected lifecycle, not a pile of independent tools.
Spotify · Data & platformThe winding road through TFX and Kubeflow
Platform adoption is a product and socio-technical problem.
Netflix · Data & platformMetaflow: a human-centric path from notebook to production
The best abstraction preserves the user’s reasoning model while adding production guarantees.
Uber · Training & distributionHorovod: make distributed training feel small
Distributed training performance is the overlap of compute, communication, input, and synchronization.
OpenAI · Training & distributionScaling Kubernetes to 7,500 nodes
At scale, metadata and coordination become a workload of their own.
NVIDIA · Training & distributionMegatron: composing parallelism for giant models
Parallelism is a placement problem constrained by bytes, shapes, and topology.
PyTorch · Training & distributionPerformant distributed checkpointing
A checkpoint is useful only when it is committed, portable, and routinely restored.
PyTorch · Training & distributionCompiling production training models safely
Compiler wins are workload-specific artifacts that need the same release discipline as models.
Google · Fairness, privacy & securityPrivate training for on-device language models
Privacy is an end-to-end guarantee with explicit adversary, accounting, and utility tradeoffs.