What we can attribute directly
Uber introduced Horovod as a distributed training framework based on ring all-reduce.
The publication describes scaling TensorFlow training with limited application changes.
Read Uber Engineering — Horovod Primary source · last checked 26 Aug 2026The operating envelope
Communication bandwidth, framework integration, cluster scheduling, gradient correctness, and developer usability.
Model teams, platform owners, operators, downstream product systems, and people affected by decisions.
Versioned data, configs, traces, artifacts, deployments, and outcomes aligned on one timeline.
Adding workers produces weak or negative speedup because synchronization dominates useful compute.
Trace the system before naming the bug.
- 01
Producers emit versioned data or model artifacts.
- 02
A platform validates, computes, stores, schedules, or routes them.
- 03
Training or inference consumes the exact declared version.
- 04
Telemetry joins the decision to system, data, and model identity.
- 05
Operators compare outcomes, stop conditions, and the last known-good path.
Follow the evidence boundary by boundary.
Symptoms
Adding workers produces weak or negative speedup because synchronization dominates useful compute.
Investigation
Profile step compute, collective time, input stalls, stragglers, tensor sizes, and scaling efficiency.
Eight workers achieve only 2.2× speedup. Which timeline measurements separate network, dataloader, and straggler causes?
Open investigation scaffold
- Write the earliest known-bad timestamp.
- Compare exact identities on either side of that boundary.
- Find the smallest affected slice and a known-good counterexample.
- Separate mitigation from root-cause confirmation.
Repair the contract, not only the symptom.
Communication, skew, and input bottlenecks violate the assumption that work scales evenly.
Use efficient collectives, fuse small tensors, balance workers, and increase compute per synchronization where valid.
Rollout
Prove numerical parity on two workers, scale gradually, and checkpoint in a topology-independent format.
Reason about the tempting shortcuts.
- Counting GPUs as a performance metric.
- Changing batch semantics without revalidating optimization.
Make recurrence visible early.
Scaling efficiency
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Collective and input wait time
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Samples per second and convergence
Define owner, slice, normal range, alert persistence, and the exact mitigation the alert should trigger.
Distributed training performance is the overlap of compute, communication, input, and synchronization.
Carry this pattern into assignments as a design constraint and into incident reviews as a hypothesis—not as proof about an unpublished system.