Operators, product owners, and people affected by the serving systems decision.
Production · weeks 22–23
Serving & inference systems
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
Opening production brief
A measurable problem, before a model.
prediction contracts, REST/gRPC, batching, timeouts with version, owner, event time, and availability contracts.
Launch traffic causes p99 spikes and leaves expensive GPUs idle. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Batch and online paths with batching, autoscaling, load tests, canaries, and rollback.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
Choose batch, online, streaming, edge, or hybrid
Place inference where the decision’s deadline, freshness, and failure semantics can actually be met.
65–90 min · checkpoint · deterministic lab - 02↗
Package model, preprocessing, schema, and runtime
The deployable artifact is the complete raw-input-to-output program, not the weights alone.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
Features, batching, caching, and backpressure
Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought.
65–90 min · checkpoint · deterministic lab - 04↗
Capacity, load testing, autoscaling, and tails
Design capacity around the tail and the delay of the scaling loop, not around average throughput.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Shadow, canary, rollback, fallback, observability
A release is safe only when exposure is attributable, bounded, observable, and quickly reversible.
65–90 min · checkpoint · deterministic lab - HW↗
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
Rubric · staged hints · reference resolution
Prerequisite guidance
3 concepts make this faster.
Prerequisites are guidance, never hard gates.