Inference
Inference in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Autoregressive decoding policies
Use autoregressive decoding policies to move the efficient inference production brief toward a defensible release.
Frontier architectures & efficient inferenceKV caches, GQA/MQA, and long context
Use kv caches, gqa/mqa, and long context to move the efficient inference production brief toward a defensible release.
Frontier architectures & efficient inferenceQuantization, compilation, and fused kernels
Use quantization, compilation, and fused kernels to move the efficient inference production brief toward a defensible release.
Frontier architectures & efficient inferenceContinuous batching and speculative decoding
Use continuous batching and speculative decoding to move the efficient inference production brief toward a defensible release.
Frontier architectures & efficient inferenceDistributed and MoE inference
Use distributed and moe inference to move the efficient inference production brief toward a defensible release.
Serving & inference systemsChoose batch, online, streaming, edge, or hybrid
Use choose batch, online, streaming, edge, or hybrid to move the serving systems production brief toward a defensible release.
Serving & inference systemsPackage model, preprocessing, schema, and runtime
Use package model, preprocessing, schema, and runtime to move the serving systems production brief toward a defensible release.
Serving & inference systemsFeatures, batching, caching, and backpressure
Use features, batching, caching, and backpressure to move the serving systems production brief toward a defensible release.
Serving & inference systemsCapacity, load testing, autoscaling, and tails
Use capacity, load testing, autoscaling, and tails to move the serving systems production brief toward a defensible release.
Serving & inference systemsShadow, canary, rollback, fallback, observability
Use shadow, canary, rollback, fallback, observability to move the serving systems production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Frontier architectures & efficient inference
A capacity plan using efficient attention, quantization, batching, and modern routing.
Assignment: LLM inference capacity planCourse 18Serving & inference systems
Batch and online paths with batching, autoscaling, load tests, canaries, and rollback.
Assignment: Production inference contractCasebook
Reported facts and course reconstructions.
A gigascale feature store with Redis
Online feature serving is a latency-critical database product with ML semantics.
Shopify · Serving & costA platform for real-time predictions
Online inference is a user-facing distributed system with a model inside.
Pinterest · Serving & costGPU-accelerated inference at Pinterest
Accelerators pay off only when the entire request path feeds them efficiently.
Pinterest · Serving & costRay batch inference: coordinating data and model work
Batch inference is a bounded dataflow whose correctness includes exactly-once publication.
Meta · Serving & costTail utilization in ads inference
Distributed inference capacity is constrained by the busiest relevant slice, not the average box.
LinkedIn · Serving & costBuilding a shared GenAI application stack
A GenAI platform manages evidence and behavior across many versioned layers.
Uber · Drift & incidentsRaising the bar on ML deployment safety
ML deployment safety is progressive evidence plus rapid reversibility.