Capacity, load testing, autoscaling, and tails
Design capacity around the tail and the delay of the scaling loop, not around average throughput.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This chapter moves from model the workload to test the control loop. We will make the idea concrete in a small code example before placing it back inside a production system with p99 under the product timeout.
Include cold starts, warmup, scale-up delay, scale-down safety, dependency slowdown, and sudden bursts in the acceptance test.
Verify the behavior illustrated in capacity_report.py, then measure decision utility, slice reliability, and system latency.
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
An average latency of 40 ms can coexist with users waiting seconds. Capacity planning is about arrival variance, bottlenecks, warmup, and tail behavior—not just dividing average requests per second by average worker throughput.
Utilization below one means a stable queue only under simplifying assumptions. Bursts, long requests, dependencies, pauses, and mixed workloads create tails well before theoretical saturation. Little’s law connects average work in the system to arrival rate and time, making hidden queue growth visible when latency rises.
Load tests should replay representative concurrency, request sizes, cache states, failures, and ramp patterns. Autoscaling must watch a signal near the constrained resource—queue age, active tokens, GPU memory, or in-flight work—and account for startup and warmup delay. Keep headroom so new capacity arrives before the SLO is already lost.
- autoscaling
- readiness
- canaries
- fallback
Follow the reasoning before touching the code.
- 01
Model the workload
Measure arrival distribution, request mix, service time, cancellation, cache hit rate, and dependency behavior from production-shaped traces.
- 02
Find the knee
Increase load gradually and record p50, p95, p99, throughput, errors, queue age, utilization, and resource saturation.
- 03
Test the control loop
Include cold starts, warmup, scale-up delay, scale-down safety, dependency slowdown, and sudden bursts in the acceptance test.
import numpy as np
latency_ms = np.array([22, 24, 25, 27, 31, 45, 80, 210, 480, 900])
arrival_rps = 50
for percentile in (50, 95, 99):
value = np.percentile(latency_ms, percentile)
print(f'p{percentile}: {value:.1f} ms')
mean_seconds = latency_ms.mean() / 1000
print('mean in system:', arrival_rps * mean_seconds)Report tails and queue growth instead of one average
What to notice
- The mean hides the small group of very slow requests that dominates p95 and p99.
- Little’s law estimates average concurrent work; a rising value at fixed arrival rate signals that requests are spending longer queued or executing.
YOUR TURN Pause and predict before revealing the answer.
A replica needs 90 seconds to load and warm. Why is scaling only after the queue violates its SLO too late?
Need a hint?
The queued requests keep aging while the new replica is unavailable.
Check your reasoning
Capacity arrives after at least another 90 seconds of accumulation. Scale on a leading signal with headroom, maintain warm capacity, or shed load before the queue crosses the SLO.
Lessons 1, 2, 3 in this course.
Connect the chapter to the course’s quantitative spine.
Design capacity around the tail and the delay of the scaling loop, not around average throughput. The queue utilization and little’s law below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Design capacity around the tail and the delay of the scaling loop, not around average throughput. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Model the workload” into a written invariant and a failing test.
- 02
Promote capacity_report.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Test the control loop” as the release checkpoint: Include cold starts, warmup, scale-up delay, scale-down safety, dependency slowdown, and sudden bursts in the acceptance test.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.