Operators, product owners, and people affected by the serving systems decision.
Production · weeks 22–23
Serving & inference systems
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
Opening production brief
A measurable problem, before a model.
prediction contracts, REST/gRPC, batching, timeouts with version, owner, event time, and availability contracts.
Launch traffic causes p99 spikes and leaves expensive GPUs idle. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Batch and online paths with batching, autoscaling, load tests, canaries, and rollback.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
Choose batch, online, streaming, edge, or hybrid
Use choose batch, online, streaming, edge, or hybrid to move the serving systems production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 02↗
Package model, preprocessing, schema, and runtime
Use package model, preprocessing, schema, and runtime to move the serving systems production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
Features, batching, caching, and backpressure
Use features, batching, caching, and backpressure to move the serving systems production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 04↗
Capacity, load testing, autoscaling, and tails
Use capacity, load testing, autoscaling, and tails to move the serving systems production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Shadow, canary, rollback, fallback, observability
Use shadow, canary, rollback, fallback, observability to move the serving systems production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - HW↗
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
Rubric · staged hints · reference resolution
Prerequisite guidance
3 concepts make this faster.
Prerequisites are guidance, never hard gates.