Drift, retraining, rollback, continuous verification
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need.
Model quality decays while every infrastructure dashboard stays green.
This chapter moves from classify the signal to retrain through release gates. We will make the idea concrete in a small code example before placing it back inside a production system with release error budget and label maturity.
Validate snapshots, compare a challenger with the champion, shadow or canary it, and retain tested rollback and fallback paths.
Verify the behavior illustrated in retrain_gate.py, then measure decision utility, slice reliability, and system latency.
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A changed input distribution is a signal to investigate, not an automatic command to retrain. Drift may reflect a real population shift, a harmless seasonal pattern, a new product policy, or a broken pipeline—and retraining on corrupted data can amplify the problem.
Data drift compares input distributions; prediction drift compares outputs; concept drift means the relationship between inputs and outcomes changed. Only mature labels can directly reveal the last case. Monitor slices and time windows against meaningful baselines, then connect the change to decision performance and operational context.
A retraining policy should require valid data, label maturity, reproducible code, comparative evaluation, and release gates. Champion-challenger evaluation, replay tests, shadowing, and rollback keep continuous learning from becoming continuous unreviewed deployment. Retain the exact previous bundle and compatible features until reversal is proven.
- guardrails
- champion/challenger
Follow the reasoning before touching the code.
- 01
Classify the signal
Locate whether the change is in data, availability, predictions, policy, outcomes, or instrumentation and which slices are affected.
- 02
Test the need to learn
Wait for appropriate labels or use a controlled evaluation to determine whether the current decision function has actually degraded.
- 03
Retrain through release gates
Validate snapshots, compare a challenger with the champion, shadow or canary it, and retain tested rollback and fallback paths.
def retrain_decision(drift, data_valid, labels_mature, quality_drop):
if not data_valid:
return 'quarantine data and investigate'
if drift and not labels_mature:
return 'monitor; wait for outcome evidence'
if labels_mature and quality_drop:
return 'train challenger through release gates'
return 'keep champion'
print(retrain_decision(True, True, False, False))Drift alone cannot authorize a new model
What to notice
- The same drift alert leads to different actions depending on data validity and outcome maturity.
- Training a challenger is not the same as promoting it; comparative evidence and rollout controls still follow.
YOUR TURN Pause and predict before revealing the answer.
Input drift is large, but calibration and decision cost remain stable on mature labels. What is the default action?
Need a hint?
Ask whether the monitored change has harmed the decision contract.
Check your reasoning
Keep the champion, continue sliced monitoring, and investigate the source. Retraining adds risk without evidence that the current decision has degraded.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need. The difference-in-means experiment estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Difference-in-means experiment estimate
| Symbol | Meaning / shape / unit |
|---|---|
Ȳ_T, Ȳ_C | treatment/control means |
s² | sample variance |
n | independent randomized units |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Ȳ_T, Ȳ_C, s², n.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c19(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Drift triage lab
Increase shift magnitude and compare statistical signal with estimated business impact.
Assumption: Ten-thousand observations; model sensitivity fixed at 0.18 utility points per σ.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| global | 0.60σ | 10.8% |
| new users | 0.67σ | 12.7% |
| mobile | 0.74σ | 14.7% |
| region A | 0.82σ | 16.6% |
| region B | 0.89σ | 18.6% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Continuous verification should automate evidence collection; promotion remains a gated response to demonstrated decision need. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Classify the signal” into a written invariant and a failing test.
- 02
Promote retrain_gate.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Retrain through release gates” as the release checkpoint: Validate snapshots, compare a challenger with the champion, shadow or canary it, and retain tested rollback and fallback paths.
- 04
Serve or materialize behind explicit release error budget and label maturity.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
- Experiment validity report
- Layered reliability diagnosis
- SLO and runbook
- Postmortem/retraining policy
Read primary material with a purpose.
Return to the opening failure.
Every release needs causal evidence, multi-layer observability, tested rollback, and a retraining policy that does not automate faults.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.