concept:replay-tests
Replay Tests
Replay Tests in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Reliability, monitoring & experimentation
Diagnose, mitigate, and write postmortems
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story.
Reliability, monitoring & experimentationRandomized experiments and guardrails
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree.
Reliability, monitoring & experimentationDrift, retraining, rollback, continuous verification
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need.
Courses & assignments
Dependency-authoritative learning units.
Casebook
Reported facts and course reconstructions.