ML platform
Infrastructure and paved roads that let teams train, ship, observe, and govern models safely.
Lessons
Formal explanations, examples, simulations, and checkpoints.
From notebook to reproducible run
Use from notebook to reproducible run to move the engineering foundations production brief toward a defensible release.
Engineering foundationsContracts before models
Use contracts before models to move the engineering foundations production brief toward a defensible release.
Engineering foundationsTraining code is product code
Use training code is product code to move the engineering foundations production brief toward a defensible release.
Engineering foundationsContainers and configuration without surprises
Use containers and configuration without surprises to move the engineering foundations production brief toward a defensible release.
Engineering foundationsObservable, idempotent, failure-safe workflows
Use observable, idempotent, failure-safe workflows to move the engineering foundations production brief toward a defensible release.
Engineering foundationsRead a research paper as an executable specification
Turn a research paper into a testable claim map, implementation contract, and prioritized reading plan before writing code.
Engineering foundationsReproduce, validate, and productionize a paper
Implement a paper from baseline to novel component, diagnose reproduction gaps, and adapt it for production without silently changing its claim.
Data & feature foundationsData contracts, ownership, and lineage
Use data contracts, ownership, and lineage to move the data & features production brief toward a defensible release.
Data & feature foundationsPoint-in-time-correct features
Use point-in-time-correct features to move the data & features production brief toward a defensible release.
Data & feature foundationsFeature transformations that survive production
Use feature transformations that survive production to move the data & features production brief toward a defensible release.
Data & feature foundationsBatch, stream, and training-serving parity
Use batch, stream, and training-serving parity to move the data & features production brief toward a defensible release.
Data & feature foundationsLabels, snapshots, and backfills
Use labels, snapshots, and backfills to move the data & features production brief toward a defensible release.
Distributed & fault-tolerant trainingChoose data, sharded, tensor, or pipeline parallelism
Use choose data, sharded, tensor, or pipeline parallelism to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDDP under the hood
Use ddp under the hood to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingFSDP and ZeRO-style sharding
Use fsdp and zero-style sharding to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingDistributed checkpoints and fault-tolerant restarts
Use distributed checkpoints and fault-tolerant restarts to move the distributed training production brief toward a defensible release.
Distributed & fault-tolerant trainingHangs, stragglers, and scaling efficiency
Use hangs, stragglers, and scaling efficiency to move the distributed training production brief toward a defensible release.
Production data & ML platformsData contracts, ownership, lineage, and quality
Use data contracts, ownership, lineage, and quality to move the ml platforms production brief toward a defensible release.
Production data & ML platformsLakehouse, batch, streaming, and event time
Use lakehouse, batch, streaming, and event time to move the ml platforms production brief toward a defensible release.
Production data & ML platformsFeature platforms and point-in-time correctness
Use feature platforms and point-in-time correctness to move the ml platforms production brief toward a defensible release.
Production data & ML platformsOrchestration, metadata, experiments, and registries
Use orchestration, metadata, experiments, and registries to move the ml platforms production brief toward a defensible release.
Production data & ML platformsPaved roads, multi-tenancy, and cost boundaries
Use paved roads, multi-tenancy, and cost boundaries to move the ml platforms production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Engineering foundations
A seeded, tested, packaged, versioned training workflow with a model card.
Assignment: Reproducible baseline packageCourse 03Data & feature foundations
Data contracts, point-in-time features, backfills, quality gates, and lineage.
Assignment: Point-in-time feature systemCourse 13Distributed & fault-tolerant training
Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.
Assignment: Fault-tolerant distributed trainerCourse 17Production data & ML platforms
A traced batch/stream platform with orchestration, registry, lineage, and ownership.
Assignment: Local mini ML platformCasebook
Reported facts and course reconstructions.
Michelangelo: one ML platform across the lifecycle
A useful ML platform is a connected lifecycle, not a pile of independent tools.
Airbnb · Data & platformChronon: feature definitions across offline and online paths
The feature definition is the product; stores and compute engines are implementations.
DoorDash · Data & platformA gigascale feature store with Redis
Online feature serving is a latency-critical database product with ML semantics.
Spotify · Data & platformThe winding road through TFX and Kubeflow
Platform adoption is a product and socio-technical problem.
Netflix · Data & platformMetaflow: a human-centric path from notebook to production
The best abstraction preserves the user’s reasoning model while adding production guarantees.
OpenAI · Training & distributionScaling Kubernetes to 7,500 nodes
At scale, metadata and coordination become a workload of their own.
Shopify · Serving & costA platform for real-time predictions
Online inference is a user-facing distributed system with a model inside.
Pinterest · Serving & costRay batch inference: coordinating data and model work
Batch inference is a bounded dataflow whose correctness includes exactly-once publication.
LinkedIn · Fairness, privacy & securityA fairness toolkit for large-scale AI
Fairness infrastructure must be context-aware, governed, and integrated into decisions.