Observe system, data, prediction, and business
Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice.
Model quality decays while every infrastructure dashboard stays green.
This chapter moves from instrument boundaries to compare meaningful baselines. We will make the idea concrete in a small code example before placing it back inside a production system with release error budget and label maturity.
Alert on sustained, sliced deviations from seasonal or release baselines and link every alert to a diagnostic query.
Verify the behavior illustrated in prediction_event.json, then measure decision utility, slice reliability, and system latency.
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
No single dashboard explains an ML decision. CPU can be healthy while data is late; data can be valid while predictions collapse; predictions can look stable while the business changes. Diagnosis requires correlated evidence across layers.
System telemetry covers traffic, errors, latency, saturation, queues, and dependencies. Data telemetry covers schema, freshness, missingness, ranges, categories, and lineage. Prediction telemetry covers score and decision distributions, calibration proxies, abstention, and slices. Business telemetry covers actions, costs, complaints, and mature outcomes.
The join keys matter more than another chart. Log request, model, transform, feature, policy, experiment, and cohort identities with privacy-aware retention. Baselines should reflect seasonality and known segments; otherwise normal weekly changes look like drift and a localized failure disappears inside a global average.
- delayed labels
- drift
- replay tests
- randomization units
Follow the reasoning before touching the code.
- 01
Instrument boundaries
Emit a small set of trusted measurements at request, feature, model, policy, and outcome boundaries.
- 02
Preserve correlation
Carry version and cohort identifiers so a symptom can be traced across layers without logging unnecessary raw sensitive data.
- 03
Compare meaningful baselines
Alert on sustained, sliced deviations from seasonal or release baselines and link every alert to a diagnostic query.
{
"request_id": "req_4d1c",
"model_version": "risk_v42",
"feature_snapshot": "fs_2026_08_29_10_00",
"policy_version": "review_v7",
"experiment": "risk_canary:treatment",
"slice": {"region": "west", "channel": "mobile"},
"score": 0.83,
"decision": "review",
"latency_ms": 47
}A privacy-aware event keeps the layers joinable
What to notice
- The event can be joined to later outcomes by request identity and attributed to the exact model, data, policy, and experiment.
- It records coarse operational slices instead of copying raw user content into a long-lived telemetry store.
YOUR TURN Pause and predict before revealing the answer.
Scores shift only for mobile traffic in one region after a release. Which comparison narrows the cause fastest?
Need a hint?
Hold cohort constant and compare version exposure plus upstream feature behavior.
Check your reasoning
Compare baseline and candidate versions within that mobile-region slice, then inspect feature snapshot, missingness, schema, and policy changes for the same requests before blaming global drift.
Lessons 1 in this course.
Connect the chapter to the course’s quantitative spine.
Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice. The difference-in-means experiment estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Difference-in-means experiment estimate
| Symbol | Meaning / shape / unit |
|---|---|
Ȳ_T, Ȳ_C | treatment/control means |
s² | sample variance |
n | independent randomized units |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Ȳ_T, Ȳ_C, s², n.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c19(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Drift triage lab
Increase shift magnitude and compare statistical signal with estimated business impact.
Assumption: Ten-thousand observations; model sensitivity fixed at 0.18 utility points per σ.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| global | 0.60σ | 10.8% |
| new users | 0.67σ | 12.7% |
| mobile | 0.74σ | 14.7% |
| region A | 0.82σ | 16.6% |
| region B | 0.89σ | 18.6% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Observability becomes diagnosis only when every layer can be joined by exact version, request, time, and slice. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Instrument boundaries” into a written invariant and a failing test.
- 02
Promote prediction_event.json from illustration to a versioned implementation with representative fixtures.
- 03
Use “Compare meaningful baselines” as the release checkpoint: Alert on sustained, sliced deviations from seasonal or release baselines and link every alert to a diagnostic query.
- 04
Serve or materialize behind explicit release error budget and label maturity.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
- Experiment validity report
- Layered reliability diagnosis
- SLO and runbook
- Postmortem/retraining policy
Read primary material with a purpose.
Return to the opening failure.
Every release needs causal evidence, multi-layer observability, tested rollback, and a retraining policy that does not automate faults.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.