Latency
Latency in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Choose batch, online, streaming, edge, or hybrid
Place inference where the decision’s deadline, freshness, and failure semantics can actually be met.
Serving & inference systemsPackage model, preprocessing, schema, and runtime
The deployable artifact is the complete raw-input-to-output program, not the weights alone.
Serving & inference systemsFeatures, batching, caching, and backpressure
Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought.
Serving & inference systemsCapacity, load testing, autoscaling, and tails
Design capacity around the tail and the delay of the scaling loop, not around average throughput.
Serving & inference systemsShadow, canary, rollback, fallback, observability
A release is safe only when exposure is attributable, bounded, observable, and quickly reversible.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.
Scaling Kubernetes to 7,500 nodes
At scale, metadata and coordination become a workload of their own.
Shopify · Serving & costA platform for real-time predictions
Online inference is a user-facing distributed system with a model inside.
Meta · Serving & costTail utilization in ads inference
Distributed inference capacity is constrained by the busiest relevant slice, not the average box.