What we can attribute directly
Uber published practices for improving ML model deployment safety.
The work covers validation and safer promotion of model versions.
Read Uber Engineering — Deployment safety Primary source · last checked 26 Aug 2026The operating envelope
Many models, delayed outcomes, high deployment rate, heterogeneous risks, and fast rollback needs.
Model teams, platform owners, operators, downstream product systems, and people affected by decisions.
Versioned data, configs, traces, artifacts, deployments, and outcomes aligned on one timeline.
Offline-successful models create online regressions because packaging, features, traffic, or unseen slices differ.
Trace the system before naming the bug.
- 01
Producers emit versioned data or model artifacts.
- 02
A platform validates, computes, stores, schedules, or routes them.
- 03
Training or inference consumes the exact declared version.
- 04
Telemetry joins the decision to system, data, and model identity.
- 05
Operators compare outcomes, stop conditions, and the last known-good path.
Follow the evidence boundary by boundary.
Symptoms
Offline-successful models create online regressions because packaging, features, traffic, or unseen slices differ.
Investigation
Compare artifact lineage, golden requests, shadow outputs, canary slices, system health, and available proxy outcomes.
Labels mature in seven days. Propose three earlier canary guardrails and the rollback rule.
Open investigation scaffold
- Write the earliest known-bad timestamp.
- Compare exact identities on either side of that boundary.
- Find the smallest affected slice and a known-good counterexample.
- Separate mitigation from root-cause confirmation.
Repair the contract, not only the symptom.
A model is promoted as a file rather than a complete decision-system release.
Automate static checks, artifact parity, shadowing, canary analysis, guardrails, and one-step rollback.
Rollout
Increase exposure only when each gate passes; retain the prior model and feature contract.
Reason about the tempting shortcuts.
- Full traffic immediately after offline evaluation.
- Manual approval without machine-checkable evidence.
Make recurrence visible early.
Canary delta by slice
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Serving error/latency
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Proxy and mature outcome guardrails
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
ML deployment safety is progressive evidence plus rapid reversibility.
Carry this pattern into assignments as a design constraint and into incident reviews as a hypothesis—not as proof about an unpublished system.