Operators, product owners, and people affected by the reliability decision.
Production · weeks 23
Reliability, monitoring & experimentation
Model quality decays while every infrastructure dashboard stays green.
Opening production brief
A measurable problem, before a model.
SLIs/SLOs, delayed labels, drift, replay tests with version, owner, event time, and availability contracts.
Model quality decays while every infrastructure dashboard stays green. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: Joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.
Frame and baseline
Turn the production problem into explicit data, metric, and baseline contracts.
- 01↗
ML SLIs, SLOs, error budgets, ownership
ML reliability joins immediate system health with delayed decision truth under one owned operating contract.
65–90 min · checkpoint · deterministic lab - 02↗
Observe system, data, prediction, and business
Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice.
65–90 min · checkpoint · deterministic lab
Build and stress
Formalize the machinery, test counterexamples, and expose system limits.
- 03↗
Diagnose, mitigate, and write postmortems
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story.
65–90 min · checkpoint · deterministic lab - 04↗
Randomized experiments and guardrails
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree.
65–90 min · checkpoint · deterministic lab
Resolve and operate
Join model behavior to architecture, observability, rollout, and rollback.
- 05↗
Drift, retraining, rollback, continuous verification
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need.
65–90 min · checkpoint · deterministic lab - HW↗
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
Rubric · staged hints · reference resolution
Prerequisite guidance
3 concepts make this faster.
Prerequisites are guidance, never hard gates.
C02EvaluationA metric contract, leakage-safe validation, calibration analysis, and experiment plan.C17ML platformsA traced batch/stream platform with orchestration, registry, lineage, and ownership.C18Serving systemsBatch and online paths with batching, autoscaling, load tests, canaries, and rollback.