Drift
Drift in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Forecast contracts and naive models
A forecast begins with two clocks—when you know and when you predict—and a baseline that respects both.
Forecasting & temporal MLTrend, seasonality, and known-future signals
A temporal feature is valid only if its value could be reconstructed exactly at the forecast origin.
Forecasting & temporal MLBacktesting without time travel
A faithful backtest is a replay of past information states, not merely a sorted split.
Forecasting & temporal MLStatistical and ML forecasters
Choose a forecaster and a multi-step strategy together; the estimator alone does not define the forecast.
Forecasting & temporal MLIntervals, drift, and forecast operations
Forecast reliability is the joint discipline of calibrated ranges, drift evidence, and a usable fallback.
Reliability, monitoring & experimentationML SLIs, SLOs, error budgets, ownership
ML reliability joins immediate system health with delayed decision truth under one owned operating contract.
Reliability, monitoring & experimentationObserve system, data, prediction, and business
Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice.
Reliability, monitoring & experimentationDiagnose, mitigate, and write postmortems
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story.
Reliability, monitoring & experimentationRandomized experiments and guardrails
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree.
Reliability, monitoring & experimentationDrift, retraining, rollback, continuous verification
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need.
Courses & assignments
Dependency-authoritative learning units.
Forecasting & temporal ML
Rolling validation, statistical baselines, boosted forecasts, and uncertainty bands.
Assignment: Regional demand forecastCourse 19Reliability, monitoring & experimentation
Joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.
Assignment: Launch analysis and incident responseCasebook
Reported facts and course reconstructions.
D3: automated data-drift detection
Drift detection is triage; impact and diagnosis decide the action.
Cloudflare · Drift & incidentsMonitoring ML models for bot detection
Adversarial ML requires joined telemetry and active probes, not passive averages.
Stripe · Fairness, privacy & securityStripe Radar: learning fraud under adversarial pressure
Risk modeling connects delayed supervision, adversarial drift, calibrated decisions, and layered controls.