Operators, product owners, and people affected by the reliability decision.
Production · weeks 23
Reliability, monitoring & experimentation
Model quality decays while every infrastructure dashboard stays green.
Opening production brief
A measurable problem, before a model.
SLIs/SLOs, delayed labels, drift, replay tests with version, owner, event time, and availability contracts.
Model quality decays while every infrastructure dashboard stays green. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
ML SLIs, SLOs, error budgets, ownership
Use ml slis, slos, error budgets, ownership to move the reliability production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 02↗
Observe system, data, prediction, and business
Use observe system, data, prediction, and business to move the reliability production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
Diagnose, mitigate, and write postmortems
Use diagnose, mitigate, and write postmortems to move the reliability production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 04↗
Randomized experiments and guardrails
Use randomized experiments and guardrails to move the reliability production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Drift, retraining, rollback, continuous verification
Use drift, retraining, rollback, continuous verification to move the reliability production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - HW↗
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
Rubric · staged hints · reference resolution
Prerequisite guidance
3 concepts make this faster.
Prerequisites are guidance, never hard gates.
C02EvaluationA metric contract, leakage-safe validation, calibration analysis, and experiment plan.C17ML platformsA traced batch/stream platform with orchestration, registry, lineage, and ownership.C18Serving systemsBatch and online paths with batching, autoscaling, load tests, canaries, and rollback.