What we can attribute directly
Uber describes Michelangelo as an internal end-to-end system for building and deploying ML models at scale.
The published architecture includes a feature store, model repository, offline training, online prediction, and monitoring.
Read Uber Engineering — Michelangelo Primary source · last checked 26 Aug 2026The operating envelope
Many teams, heterogeneous use cases, shared infrastructure, and both batch and low-latency prediction paths.
Model teams, platform owners, operators, downstream product systems, and people affected by decisions.
Versioned data, configs, traces, artifacts, deployments, and outcomes aligned on one timeline.
Before a paved road, teams repeat integration work and struggle to reproduce features and models across environments.
Trace the system before naming the bug.
- 01
Producers emit versioned data or model artifacts.
- 02
A platform validates, computes, stores, schedules, or routes them.
- 03
Training or inference consumes the exact declared version.
- 04
Telemetry joins the decision to system, data, and model identity.
- 05
Operators compare outcomes, stop conditions, and the last known-good path.
Follow the evidence boundary by boundary.
Symptoms
Before a paved road, teams repeat integration work and struggle to reproduce features and models across environments.
Investigation
Map every artifact boundary: source data → feature computation → training dataset → run → model version → serving endpoint → observed outcome.
A prediction regresses after a feature backfill. Which identities must you join to reconstruct the exact online decision?
Open investigation scaffold
- Write the earliest known-bad timestamp.
- Compare exact identities on either side of that boundary.
- Find the smallest affected slice and a known-good counterexample.
- Separate mitigation from root-cause confirmation.
Repair the contract, not only the symptom.
Lifecycle components owned in isolation create handoff gaps, skew, and weak lineage.
Create versioned platform contracts that connect feature definitions, runs, artifacts, deployments, and monitoring.
Rollout
Adopt by one high-value workflow, retain escape hatches, and expand the paved road only after real operating feedback.
Reason about the tempting shortcuts.
- A universal black-box API that hides every model-specific need.
- A model registry without reproducible data and feature lineage.
Make recurrence visible early.
Feature freshness and parity
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Prediction latency and volume
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Model outcome quality by version
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
A useful ML platform is a connected lifecycle, not a pile of independent tools.
Carry this pattern into assignments as a design constraint and into incident reviews as a hypothesis—not as proof about an unpublished system.