Distributed and MoE inference
Distributed inference is constrained by the slowest communication or routing boundary, not by the sum of advertised compute.
Long-context generation breaks latency and cost targets.
This chapter moves from draw the communication path to define overload semantics. We will make the idea concrete in a small code example before placing it back inside a production system with p99 first-token and cost/token SLOs.
Specify what happens when an expert or rank is saturated: queue, reroute, drop, fall back, or reject—with a quality test for that choice.
Verify the behavior illustrated in expert_capacity.py, then measure decision utility, slice reliability, and system latency.
Maximizing aggregate tokens/s destroys p99 first-token latency and harms a rare-language quality slice. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
When one accelerator cannot hold or serve a model, inference becomes a communication problem. Tensor and pipeline parallelism split dense work; mixture-of-experts models activate only part of their parameters but introduce token routing and load imbalance.
Tensor parallelism shards a layer and communicates partial results inside nearly every block. Pipeline parallelism assigns groups of layers to stages and passes activations between them. The best split depends on topology, batch shape, decode frequency, and whether communication can overlap compute—not on parameter count alone.
An MoE router sends each token to a small set of experts. Sparse activation lowers arithmetic per token, yet popular experts can exceed capacity while other devices idle. Capacity factors, token dropping, expert placement, and cross-device all-to-all traffic become part of model quality and serving reliability.
- speculative decoding
- MoE
Follow the reasoning before touching the code.
- 01
Draw the communication path
For one decode token, mark every collective, activation transfer, and synchronization boundary across the physical interconnect.
- 02
Inspect imbalance
Track tokens per expert and per rank by request slice; averages hide hot experts and slow pipeline stages.
- 03
Define overload semantics
Specify what happens when an expert or rank is saturated: queue, reroute, drop, fall back, or reject—with a quality test for that choice.
from collections import Counter
routes = [0, 0, 2, 0, 1, 2, 0, 3, 0, 2]
experts, capacity = 4, 3
load = Counter(routes)
for expert in range(experts):
used = load[expert]
overflow = max(0, used - capacity)
print(expert, {'used': used, 'overflow': overflow})Expose an overloaded expert before deployment
What to notice
- The total token count fits aggregate capacity, yet expert 0 still overflows because routing is uneven.
- Adding devices does not guarantee lower latency if the chosen parallelism adds a collective to every decode step.
YOUR TURN Pause and predict before revealing the answer.
Raise capacity from three to five. Does that prove the MoE deployment is balanced?
Need a hint?
Capacity prevents one symptom, but compare the loads and communication time across experts.
Check your reasoning
No. Overflow disappears, but expert 0 still receives far more work, so its rank can remain the latency bottleneck while other experts are underused.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
Distributed inference is constrained by the slowest communication or routing boundary, not by the sum of advertised compute. The kv-cache memory estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
KV-cache memory estimate
| Symbol | Meaning / shape / unit |
|---|---|
L | decoder layers |
B | active sequences |
S | cached tokens |
H_kv | KV heads |
q | bytes per scalar |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: L, B, S, H_kv, q.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c16(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
GPU memory ledger
Change microbatch size and calculate a simplified training-memory budget.
Assumption: 3.2 GB weights+grads+optimizer, 0.34 GB activations per sample, 24 GB device, 10% reserve.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| block 1 | 1.09 GB activations | resident |
| block 2 | 2.18 GB activations | resident |
| block 3 | 3.26 GB activations | resident |
| block 4 | 4.35 GB activations | resident |
| block 5 | 5.44 GB activations | resident |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Distributed inference is constrained by the slowest communication or routing boundary, not by the sum of advertised compute. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Draw the communication path” into a written invariant and a failing test.
- 02
Promote expert_capacity.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Define overload semantics” as the release checkpoint: Specify what happens when an expert or rank is saturated: queue, reroute, drop, fall back, or reject—with a quality test for that choice.
- 04
Serve or materialize behind explicit p99 first-token and cost/token SLOs.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Maximizing aggregate tokens/s destroys p99 first-token latency and harms a rare-language quality slice. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
LLM inference capacity plan
Benchmark cache, batching, concurrency, sampling, and one quantized path on mixed traces.
- Benchmark contract
- Profiler/bottleneck report
- Quality gates
- Capacity and cost recommendation
Read primary material with a purpose.
Return to the opening failure.
Choose optimizations from workload traces with quality gates, tail latency, memory headroom, and cost per token.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a capacity plan using efficient attention, quantization, batching, and modern routing.