ML SLIs, SLOs, error budgets, ownership
ML reliability joins immediate system health with delayed decision truth under one owned operating contract.
Model quality decays while every infrastructure dashboard stays green.
This chapter moves from map the decision chain to connect budget to action. We will make the idea concrete in a small code example before placing it back inside a production system with release error budget and label maturity.
Set burn alerts, release freezes, owners, and escalation paths before an incident consumes the full window.
Verify the behavior illustrated in error_budget.py, then measure decision utility, slice reliability, and system latency.
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
An ML service can be available while making stale or harmful decisions. Its reliability contract must cover the system, the data, the predictions, and the eventual outcomes—even when labels arrive days later.
An SLI is a measured indicator such as successful low-latency responses, fresh-feature coverage, calibration error, or mature-label decision loss. An SLO gives that indicator a target over a window. Because quality labels are delayed, pair fast proxies and invariants with slower outcome measures without pretending they are interchangeable.
The error budget is the allowed unreliability implied by the SLO. Burn rate shows how quickly it is being consumed and should control release pace and incident urgency. Each SLI needs an owner, query, data source, window, exclusions, alert policy, and response action; otherwise the number cannot operate the service.
- SLIs/SLOs
- delayed labels
- drift
- replay tests
Follow the reasoning before touching the code.
- 01
Map the decision chain
Identify user-visible success and the system, data, model, and policy conditions required to produce it.
- 02
Define measurable windows
Specify numerator, denominator, slice, maturity, target, and window for every SLI and document label delay.
- 03
Connect budget to action
Set burn alerts, release freezes, owners, and escalation paths before an incident consumes the full window.
target = 0.999
requests = 2_000_000
bad = 4_600
allowed_bad = requests * (1 - target)
budget_used = bad / allowed_bad
print('allowed bad:', round(allowed_bad))
print('budget used:', f'{budget_used:.1%}')
print('release freeze:', budget_used > 1)Turn an SLO into an operational budget
What to notice
- A 99.9% target allows about 2,000 bad requests in this window, so 4,600 consumes more than the full budget.
- A pass/fail calculation is only useful when “bad” matches a user-relevant definition and exclusions are not hiding failures.
YOUR TURN Pause and predict before revealing the answer.
Labels mature after seven days. What should the team alert on during the first hour of a release?
Need a hint?
Use immediate necessary conditions and proxies, but preserve the later outcome gate.
Check your reasoning
Alert on service errors and tails, feature freshness and schema, prediction distribution and abstention, version exposure, and known slice proxies; then evaluate mature decision quality when labels arrive.
Evaluation, ML platforms, Serving systems
Connect the chapter to the course’s quantitative spine.
ML reliability joins immediate system health with delayed decision truth under one owned operating contract. The difference-in-means experiment estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Difference-in-means experiment estimate
| Symbol | Meaning / shape / unit |
|---|---|
Ȳ_T, Ȳ_C | treatment/control means |
s² | sample variance |
n | independent randomized units |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Ȳ_T, Ȳ_C, s², n.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c19(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Drift triage lab
Increase shift magnitude and compare statistical signal with estimated business impact.
Assumption: Ten-thousand observations; model sensitivity fixed at 0.18 utility points per σ.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| global | 0.60σ | 10.8% |
| new users | 0.67σ | 12.7% |
| mobile | 0.74σ | 14.7% |
| region A | 0.82σ | 16.6% |
| region B | 0.89σ | 18.6% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: ML reliability joins immediate system health with delayed decision truth under one owned operating contract. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Map the decision chain” into a written invariant and a failing test.
- 02
Promote error_budget.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Connect budget to action” as the release checkpoint: Set burn alerts, release freezes, owners, and escalation paths before an incident consumes the full window.
- 04
Serve or materialize behind explicit release error budget and label maturity.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
- Experiment validity report
- Layered reliability diagnosis
- SLO and runbook
- Postmortem/retraining policy
Read primary material with a purpose.
Return to the opening failure.
Every release needs causal evidence, multi-layer observability, tested rollback, and a retraining policy that does not automate faults.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.