Inference
Inference in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Autoregressive decoding policies
The model supplies possibilities; the decoding policy decides which possibilities can become product behavior.
Frontier architectures & efficient inferenceKV caches, GQA/MQA, and long context
KV caching removes repeated history compute by turning context length into a first-class memory budget.
Frontier architectures & efficient inferenceQuantization, compilation, and fused kernels
Optimize the measured bottleneck, and treat every numeric or execution rewrite as a new artifact that must pass quality gates.
Frontier architectures & efficient inferenceContinuous batching and speculative decoding
Throughput techniques earn their place only when they improve completed requests—not merely scheduled tokens.
Frontier architectures & efficient inferenceDistributed and MoE inference
Distributed inference is constrained by the slowest communication or routing boundary, not by the sum of advertised compute.
Serving & inference systemsChoose batch, online, streaming, edge, or hybrid
Place inference where the decision’s deadline, freshness, and failure semantics can actually be met.
Serving & inference systemsPackage model, preprocessing, schema, and runtime
The deployable artifact is the complete raw-input-to-output program, not the weights alone.
Serving & inference systemsFeatures, batching, caching, and backpressure
Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought.
Serving & inference systemsCapacity, load testing, autoscaling, and tails
Design capacity around the tail and the delay of the scaling loop, not around average throughput.
Serving & inference systemsShadow, canary, rollback, fallback, observability
A release is safe only when exposure is attributable, bounded, observable, and quickly reversible.
Courses & assignments
Dependency-authoritative learning units.
Frontier architectures & efficient inference
A capacity plan using efficient attention, quantization, batching, and modern routing.
Assignment: LLM inference capacity planCourse 18Serving & inference systems
Batch and online paths with batching, autoscaling, load tests, canaries, and rollback.
Assignment: Production inference contractCasebook
Reported facts and course reconstructions.
A gigascale feature store with Redis
Online feature serving is a latency-critical database product with ML semantics.
Shopify · Serving & costA platform for real-time predictions
Online inference is a user-facing distributed system with a model inside.
Pinterest · Serving & costGPU-accelerated inference at Pinterest
Accelerators pay off only when the entire request path feeds them efficiently.
Pinterest · Serving & costRay batch inference: coordinating data and model work
Batch inference is a bounded dataflow whose correctness includes exactly-once publication.
Meta · Serving & costTail utilization in ads inference
Distributed inference capacity is constrained by the busiest relevant slice, not the average box.
LinkedIn · Serving & costBuilding a shared GenAI application stack
A GenAI platform manages evidence and behavior across many versioned layers.
Uber · Drift & incidentsRaising the bar on ML deployment safety
ML deployment safety is progressive evidence plus rapid reversibility.