Distributed checkpoints and fault-tolerant restarts
Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery.
An eight-GPU job is slower than one GPU and loses progress on failure.
This chapter moves from inventory continuation state to practice disruptive recovery. We will make the idea concrete in a small code example before placing it back inside a production system with eight GPUs with restart under 15 minutes.
Kill a rank mid-save and mid-step, restart on the same and a different world size, and compare the resumed loss path with an uninterrupted control.
Verify the behavior illustrated in checkpoint-manifest.yaml, then measure decision utility, slice reliability, and system latency.
One rank skips a batch and misses an all-reduce, leaving every other rank hung. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A checkpoint is not a bag of model weights. It is the complete answer to: “What must every rank know to take exactly the next training step?”
A faithful restart includes model and optimizer state, scheduler and scaler state, global step, random generators, sampler epoch or data cursor, and relevant configuration. In a sharded job, those tensors may be distributed across files, so a directory is valid only when all shards and metadata belong to the same logical step.
Use a two-phase commit: ranks write to a step-scoped temporary location, synchronize, verify expected shards, then publish a small completion manifest atomically. Readers ignore checkpoints without that marker. Prefer logical distributed checkpoint formats that can reshard into a different world size; rank-numbered pickles often bind recovery to the original topology.
- FSDP
- tensor parallelism
- checkpoint resharding
- stragglers
Follow the reasoning before touching the code.
- 01
Inventory continuation state
Start from the next-step equivalence test and list everything that influences it, including RNG streams and which samples have already been consumed.
- 02
Commit the set atomically
Write immutable shards first and the completion manifest last. Include checksums, logical tensor names, step, format version, and writer world size.
- 03
Practice disruptive recovery
Kill a rank mid-save and mid-step, restart on the same and a different world size, and compare the resumed loss path with an uninterrupted control.
format_version: 2
status: complete
global_step: 18000
writer_world_size: 8
model:
logical_state: model
shards: 8
optimizer:
logical_state: adamw
shards: 8
trainer_state:
scheduler_step: 18000
grad_scaler: scaler.json
rng_states: rng/
data_cursor: data-cursor.json
checksums: sha256sums.txtA completion manifest describes one verified logical checkpoint.
What to notice
- The manifest records logical state and writer topology separately, leaving room for a reader that can reshard into a new topology.
- `status: complete` is trustworthy only if it is published after every listed artifact and checksum has been durably written.
YOUR TURN Pause and predict before revealing the answer.
A restart restores weights and optimizer tensors but begins the epoch at batch zero. Why is it not equivalent?
Need a hint?
The update sequence depends on which examples and random transformations come next.
Check your reasoning
It repeats already-consumed data and changes shuffle and augmentation order. Restore the sampler epoch/data cursor and RNG states, or explicitly document a weaker restart guarantee.
Lessons 1, 2, 3 in this course.
Connect the chapter to the course’s quantitative spine.
Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery. The scaling efficiency below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Scaling efficiency
| Symbol | Meaning / shape / unit |
|---|---|
W | worker/GPU count |
throughput_W | distributed samples per second |
throughput_1 | single-worker baseline |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: W, throughput_W, throughput_1.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c13(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
All-reduce scaling lab
Add workers and estimate delivered speedup after communication and stragglers.
Assumption: 120 ms local-compute step; ring communication and logarithmic straggler penalty.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| 1 workers | 120.3 ms | 1.0× |
| 2 workers | 125.7 ms | 1.9× |
| 4 workers | 131.4 ms | 3.7× |
| 8 workers | 137.8 ms | 7.0× |
| 16 workers | 145.6 ms | 13.2× |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Checkpoint correctness is next-step equivalence, protected by complete state, atomic publication, and rehearsed recovery. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Inventory continuation state” into a written invariant and a failing test.
- 02
Promote checkpoint-manifest.yaml from illustration to a versioned implementation with representative fixtures.
- 03
Use “Practice disruptive recovery” as the release checkpoint: Kill a rank mid-save and mid-step, restart on the same and a different world size, and compare the resumed loss path with an uninterrupted control.
- 04
Serve or materialize behind explicit eight GPUs with restart under 15 minutes.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
One rank skips a batch and misses an all-reduce, leaving every other rank hung. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Fault-tolerant distributed trainer
Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.
- Gradient/data equivalence
- 1/2/4-GPU efficiency report
- Kill-and-resume proof
- Hang/straggler runbook
Read primary material with a purpose.
Return to the opening failure.
Log per-rank data/compute/collective time, preserve complete training state, and keep a one-GPU equivalence test.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce efficient ddp, parallelism choices, elastic sampling, and resumable checkpoints.