Monitoring
Monitoring in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
ML SLIs, SLOs, error budgets, ownership
ML reliability joins immediate system health with delayed decision truth under one owned operating contract.
Reliability, monitoring & experimentationObserve system, data, prediction, and business
Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice.
Reliability, monitoring & experimentationDiagnose, mitigate, and write postmortems
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story.
Reliability, monitoring & experimentationRandomized experiments and guardrails
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree.
Reliability, monitoring & experimentationDrift, retraining, rollback, continuous verification
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.
D3: automated data-drift detection
Drift detection is triage; impact and diagnosis decide the action.
Cloudflare · Drift & incidentsWhen a generated feature file caused a global outage
Generated data can behave like executable configuration and needs release engineering.
Cloudflare · Drift & incidentsMonitoring ML models for bot detection
Adversarial ML requires joined telemetry and active probes, not passive averages.