Hangs, stragglers, and scaling efficiency
Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count.
An eight-GPU job is slower than one GPU and loses progress on failure.
This chapter moves from localize the waiting rank to build a scaling curve. We will make the idea concrete in a small code example before placing it back inside a production system with eight GPUs with restart under 15 minutes.
Measure 1, 2, 4, and more workers under a fixed global-batch or clearly adjusted optimization contract. Plot throughput, efficiency, and cost to target quality.
Verify the behavior illustrated in scaling_diagnosis.py, then measure decision utility, slice reliability, and system latency.
One rank skips a batch and misses an all-reduce, leaving every other rank hung. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Distributed training moves at the pace of the slowest required participant. One delayed rank turns everyone else’s idle time into an invisible tax.
A hang usually means ranks no longer agree on collective order, tensor shape, or whether to participate at all. A straggler reaches the correct collective late because of slow input, imbalanced work, thermal throttling, network contention, or a hidden synchronization. Both look like waiting, so diagnostics need per-rank timelines and collective sequence information.
Scaling efficiency compares W-worker throughput with W times the single-worker baseline. It falls as communication, duplicated input work, imbalance, and fixed overhead grow. Report time components and quality-equivalent global-batch settings; an impressive hardware utilization number can coexist with worse time to convergence.
- checkpoint resharding
- stragglers
Follow the reasoning before touching the code.
- 01
Localize the waiting rank
Capture per-rank timestamps for data, forward, backward, collectives, and optimizer work. Use collective debug logs to find the first mismatched or late operation.
- 02
Separate hang from slowness
Confirm whether every rank eventually reaches the same collective. A missing participant suggests control-flow mismatch; a late one points to load, data, hardware, or network variance.
- 03
Build a scaling curve
Measure 1, 2, 4, and more workers under a fixed global-batch or clearly adjusted optimization contract. Plot throughput, efficiency, and cost to target quality.
single_gpu = 420.0
workers = 4
distributed = 1390.0
efficiency = distributed / (workers * single_gpu)
rank_step_ms = [118.0, 120.0, 119.0, 151.0]
slowest = max(rank_step_ms)
idle_ms = [slowest - value for value in rank_step_ms]
print('efficiency:', round(efficiency, 3))
print('slow rank:', rank_step_ms.index(slowest))
print('idle by rank:', idle_ms)
assert 0 < efficiency <= 1Quantify scaling efficiency and the idle time induced by a slow rank.
What to notice
- Four GPUs achieve about 82.7% scaling efficiency rather than the ideal 1,680 samples per second.
- Ranks 0–2 can spend roughly 31–33 ms waiting for rank 3 each step, so the slow rank is the first investigation target.
YOUR TURN Pause and predict before revealing the answer.
Throughput doubles from four to eight GPUs, but the hourly cluster cost also doubles. What else determines whether scaling helped?
Need a hint?
Compare total time and total accelerator-hours to the same quality target.
Check your reasoning
If convergence steps and reliability are unchanged, wall time halves while accelerator-hours stay roughly flat. If larger global batch needs more steps or more failures/restarts, both cost and time-to-quality can worsen.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count. The scaling efficiency below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Scaling efficiency
| Symbol | Meaning / shape / unit |
|---|---|
W | worker/GPU count |
throughput_W | distributed samples per second |
throughput_1 | single-worker baseline |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: W, throughput_W, throughput_1.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c13(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
All-reduce scaling lab
Add workers and estimate delivered speedup after communication and stragglers.
Assumption: 120 ms local-compute step; ring communication and logarithmic straggler penalty.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| 1 workers | 120.3 ms | 1.0× |
| 2 workers | 125.7 ms | 1.9× |
| 4 workers | 131.4 ms | 3.7× |
| 8 workers | 137.8 ms | 7.0× |
| 16 workers | 145.6 ms | 13.2× |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Distributed speed is synchronized speed: diagnose the first divergence, measure the slowest rank, and scale against quality and cost—not worker count. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Localize the waiting rank” into a written invariant and a failing test.
- 02
Promote scaling_diagnosis.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Build a scaling curve” as the release checkpoint: Measure 1, 2, 4, and more workers under a fixed global-batch or clearly adjusted optimization contract. Plot throughput, efficiency, and cost to target quality.
- 04
Serve or materialize behind explicit eight GPUs with restart under 15 minutes.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
One rank skips a batch and misses an all-reduce, leaving every other rank hung. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Fault-tolerant distributed trainer
Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.
- Gradient/data equivalence
- 1/2/4-GPU efficiency report
- Kill-and-resume proof
- Hang/straggler runbook
Read primary material with a purpose.
Return to the opening failure.
Log per-rank data/compute/collective time, preserve complete training state, and keep a one-GPU equivalence test.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce efficient ddp, parallelism choices, elastic sampling, and resumable checkpoints.