Production assignment 13

Fault-tolerant distributed trainer

Specify DDP correctness, scaling experiments, worker failure, and complete checkpoint recovery.

The production problem

An eight-GPU job is slower than one GPU and loses progress on failure.

You own the technical recommendation and the evidence behind it. A reviewer must be able to reproduce the decision, identify which assumptions are estimates, and operate the failure path without asking the author.

Sample input

Versioned events or rows with entity ID, event time, availability time, payload, and mature outcome where applicable.

Expected output

A registered artifact plus evaluation report, decision policy, architecture trace, observability plan, and rollback procedure.

Targets

Beat the declared baseline without violating latency, cost, capacity, subgroup, or freshness guardrails.

Deliverables

Submit evidence another engineer can inspect.

  1. 01

    Gradient/data equivalence

    Include assumptions, identity/version fields, one success example, one counterexample, and the command or query used to verify it.

  2. 02

    1/2/4-GPU efficiency report

    Include assumptions, identity/version fields, one success example, one counterexample, and the command or query used to verify it.

  3. 03

    Kill-and-resume proof

    Include assumptions, identity/version fields, one success example, one counterexample, and the command or query used to verify it.

  4. 04

    Hang/straggler runbook

    Include assumptions, identity/version fields, one success example, one counterexample, and the command or query used to verify it.

Invariants & acceptance

The result must stay true after packaging.

  • All training inputs are knowable at the historical decision cutoff.
  • Raw-input inference after serialization matches the validated reference within declared tolerance.
  • Every artifact and decision can be joined to code, data, features, configuration, and model version.
  • A failed retry is idempotent and cannot publish partial output as complete.
  • Quality is reported by temporal, operational, and affected-user slices—not aggregate only.
  • The rollback or fallback has a tested trigger, owner, and bounded recovery time.
Self-verification protocol

Try to falsify your own recommendation.

  1. Replay one hand-calculated input and one production-shaped batch.
  2. Inject a missing, late, malformed, and out-of-distribution input.
  3. Restart at the worst possible commit boundary.
  4. Load-test the constrained path, including cold start and a failed dependency.
  5. Compare one protected or high-risk slice and document uncertainty.
  6. Have the fallback produce an auditable result before calling the system ready.
Reference production resolution

Efficient DDP, parallelism choices, elastic sampling, and resumable checkpoints.

The full reference stays behind the recorded-attempt control above. Compare architectural choices and invariants, not incidental file names.