Choose data, sharded, tensor, or pipeline parallelism
Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves.
An eight-GPU job is slower than one GPU and loses progress on failure.
This chapter moves from find what cannot fit to model the communication path. We will make the idea concrete in a small code example before placing it back inside a production system with eight GPUs with restart under 15 minutes.
Estimate collective bytes, link bandwidth, stage balance, and microbatch bubbles. Then benchmark a small topology before scaling the job.
Verify the behavior illustrated in parallelism_budget.py, then measure decision utility, slice reliability, and system latency.
One rank skips a batch and misses an all-reduce, leaving every other rank hung. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Parallelism is not one switch. It is a choice about which dimension—data, model state, individual operators, or layers—must be divided to satisfy memory and throughput constraints.
Data parallelism replicates the model and gives each worker different examples, then synchronizes gradients. It is the simplest high-throughput choice when a full replica fits. Sharded data parallelism keeps the data-parallel computation but partitions parameters, gradients, or optimizer state, trading extra communication for memory relief.
Tensor parallelism splits large operations such as a matrix multiplication across devices; pipeline parallelism assigns different layer ranges to stages and streams microbatches through them. Both are usually justified only when layers or the whole model still cannot fit after sharding. The best design begins with a memory ledger and measured communication topology, not the largest available world size.
- rank/world size
- distributed samplers
- all-reduce
- global batch
Follow the reasoning before touching the code.
- 01
Find what cannot fit
Estimate parameter, gradient, optimizer, activation, and temporary-buffer memory at the desired shape. Identify whether the blocker is total state, one oversized layer, or activations.
- 02
Choose the smallest split
Use DDP when a replica fits, sharding when total state does not, tensor parallelism when an operator is too large, and pipelines when layer groups offer natural stages.
- 03
Model the communication path
Estimate collective bytes, link bandwidth, stage balance, and microbatch bubbles. Then benchmark a small topology before scaling the job.
gpu_gb = 80
parameters_gb = 28
gradients_gb = 28
optimizer_gb = 56
activations_gb = 18
replica_gb = parameters_gb + gradients_gb + optimizer_gb + activations_gb
largest_layer_gb = 6
print('DDP fits:', replica_gb <= gpu_gb)
print('shard model state:', replica_gb > gpu_gb)
print('tensor split needed:', largest_layer_gb > gpu_gb)
print('pipeline candidate:', activations_gb > parameters_gb)
print('estimated replica GB:', replica_gb)Turn a rough memory ledger into explicit parallelism candidates.
What to notice
- This replica needs about 130 GB, so plain DDP cannot fit on an 80 GB device before temporary buffers are even counted.
- The largest layer fits comfortably, so tensor parallelism is not the first remedy; sharding replicated state is the more direct candidate.
YOUR TURN Pause and predict before revealing the answer.
Suppose sharding makes the model fit but step time is dominated by parameter all-gathers. What evidence should decide the next move?
Need a hint?
The trade depends on bytes, overlap, and the physical links between ranks.
Check your reasoning
Use per-collective traces and topology-aware benchmarks to see whether communication overlaps compute. A different shard policy, larger compute chunks, faster intra-node grouping, or fewer ranks may beat adding more parallelism.
GPU training
Connect the chapter to the course’s quantitative spine.
Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves. The scaling efficiency below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Scaling efficiency
| Symbol | Meaning / shape / unit |
|---|---|
W | worker/GPU count |
throughput_W | distributed samples per second |
throughput_1 | single-worker baseline |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: W, throughput_W, throughput_1.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c13(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
All-reduce scaling lab
Add workers and estimate delivered speedup after communication and stragglers.
Assumption: 120 ms local-compute step; ring communication and logarithmic straggler penalty.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| 1 workers | 120.3 ms | 1.0× |
| 2 workers | 125.7 ms | 1.9× |
| 4 workers | 131.4 ms | 3.7× |
| 8 workers | 137.8 ms | 7.0× |
| 16 workers | 145.6 ms | 13.2× |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Parallelize the resource that actually exceeds its budget, and stop when the added communication costs more than the split saves. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Find what cannot fit” into a written invariant and a failing test.
- 02
Promote parallelism_budget.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Model the communication path” as the release checkpoint: Estimate collective bytes, link bandwidth, stage balance, and microbatch bubbles. Then benchmark a small topology before scaling the job.
- 04
Serve or materialize behind explicit eight GPUs with restart under 15 minutes.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
One rank skips a batch and misses an all-reduce, leaving every other rank hung. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Fault-tolerant distributed trainer
Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.
- Gradient/data equivalence
- 1/2/4-GPU efficiency report
- Kill-and-resume proof
- Hang/straggler runbook
Read primary material with a purpose.
Return to the opening failure.
Log per-rank data/compute/collective time, preserve complete training state, and keep a one-GPU equivalence test.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce efficient ddp, parallelism choices, elastic sampling, and resumable checkpoints.