Diagnose, mitigate, and write postmortems
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story.
Model quality decays while every infrastructure dashboard stays green.
This chapter moves from stabilize and scope to turn learning into controls. We will make the idea concrete in a small code example before placing it back inside a production system with release error budget and label maturity.
Write a timeline and contributing factors, then assign tested preventive and detective actions with owners and dates.
Verify the behavior illustrated in incident.yaml, then measure decision utility, slice reliability, and system latency.
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
During an ML incident, the first job is to reduce harm, not prove an elegant root cause. A disciplined response separates symptom, scope, and recent change; applies the safest reversible mitigation; then uses preserved evidence to learn.
Triage begins with user impact and blast radius: which decisions, slices, versions, regions, and time windows are affected? Compare against a known-good baseline and inspect boundaries in order—request, data, feature, model, policy, and outcome. Changing several parameters at once destroys the evidence needed to localize the fault.
Mitigation can roll back, disable a feature, reduce exposure, serve a fallback, freeze a pipeline, or abstain. A blameless postmortem records the factual timeline, contributing conditions, why defenses failed, and actions that change the system. “Be more careful” is not a corrective control; a test, invariant, safer default, or ownership change is.
- replay tests
- randomization units
- SRM
- confidence intervals
Follow the reasoning before touching the code.
- 01
Stabilize and scope
Declare incident ownership, bound affected decisions, preserve logs and artifacts, and choose the lowest-risk reversible mitigation.
- 02
Localize by boundary
Replay the smallest failing request across input, feature, model, and policy versions while comparing a known-good path.
- 03
Turn learning into controls
Write a timeline and contributing factors, then assign tested preventive and detective actions with owners and dates.
incident: INC-142
impact: elevated false declines in region-west
started_at: 2026-08-29T10:12:00Z
detected_at: 2026-08-29T10:21:00Z
candidate_version: risk_v42
mitigation: rollback_to_risk_v41
evidence:
- missing merchant_age feature rose from 0.2% to 18%
- only feature_snapshot fs_2026_08_29_10_00 is affected
next_check: replay golden west-region requestsA terse incident record separates evidence from interpretation
What to notice
- The record states observed facts separately from an unproven root-cause claim.
- Rollback names an exact target, which avoids replacing one ambiguous state with another.
YOUR TURN Pause and predict before revealing the answer.
Rollback restores decisions, but the corrupted feature snapshot remains available to future jobs. Is the incident resolved?
Need a hint?
Distinguish immediate mitigation from removal of the fault and prevention of recurrence.
Check your reasoning
Impact is mitigated, not resolved. Quarantine the snapshot, repair or republish it, identify why quality gates failed, verify consumers, and add controls before closing the incident.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story. The difference-in-means experiment estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Difference-in-means experiment estimate
| Symbol | Meaning / shape / unit |
|---|---|
Ȳ_T, Ȳ_C | treatment/control means |
s² | sample variance |
n | independent randomized units |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Ȳ_T, Ȳ_C, s², n.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c19(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Drift triage lab
Increase shift magnitude and compare statistical signal with estimated business impact.
Assumption: Ten-thousand observations; model sensitivity fixed at 0.18 utility points per σ.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| global | 0.60σ | 10.8% |
| new users | 0.67σ | 12.7% |
| mobile | 0.74σ | 14.7% |
| region A | 0.82σ | 16.6% |
| region B | 0.89σ | 18.6% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Mitigate harm quickly, preserve causality carefully, and make the postmortem change the system rather than the story. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Stabilize and scope” into a written invariant and a failing test.
- 02
Promote incident.yaml from illustration to a versioned implementation with representative fixtures.
- 03
Use “Turn learning into controls” as the release checkpoint: Write a timeline and contributing factors, then assign tested preventive and detective actions with owners and dates.
- 04
Serve or materialize behind explicit release error budget and label maturity.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
- Experiment validity report
- Layered reliability diagnosis
- SLO and runbook
- Postmortem/retraining policy
Read primary material with a purpose.
Return to the opening failure.
Every release needs causal evidence, multi-layer observability, tested rollback, and a retraining policy that does not automate faults.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.