Features, batching, caching, and backpressure
Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This chapter moves from budget the request path to fail before saturation. We will make the idea concrete in a small code example before placing it back inside a production system with p99 under the product timeout.
Bound queues and retries, expose remaining capacity, and return a tested fallback or overload response while the system is still controllable.
Verify the behavior illustrated in admission.py, then measure decision utility, slice reliability, and system latency.
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Most serving failures occur around the model: a slow feature lookup, an unbounded batch wait, a stale cache, or a queue that accepts work faster than the system can finish it. These controls are coupled, so every speedup needs explicit correctness and overload semantics.
Dynamic batching amortizes fixed work but adds queue delay; choose a maximum batch size and maximum wait tied to the latency budget. Caches reduce repeated computation only when keys include every input and version that affects the result, and their TTL must match freshness requirements. Feature fetches need deadlines and a defined response to missing or stale values.
Backpressure protects the service by bounding outstanding work. When capacity is exhausted, reject, shed, prioritize, or degrade early instead of allowing queues and retries to consume all memory. Coordinate retry budgets across clients and services so one slow dependency does not create an amplification storm.
- circuit breakers
- queueing
- autoscaling
- readiness
Follow the reasoning before touching the code.
- 01
Budget the request path
Allocate the end-to-end SLO across queueing, features, preprocessing, inference, postprocessing, and network overhead.
- 02
Version acceleration state
Include model, feature, transform, and policy versions in batch compatibility and cache keys; define maximum staleness.
- 03
Fail before saturation
Bound queues and retries, expose remaining capacity, and return a tested fallback or overload response while the system is still controllable.
from queue import Full, Queue
pending = Queue(maxsize=3)
def admit(request_id):
try:
pending.put_nowait(request_id)
return {'status': 'queued'}
except Full:
return {'status': 'fallback', 'reason': 'overloaded'}
for request in range(5):
print(request, admit(request))A bounded queue makes overload an explicit response
What to notice
- The fourth and fifth requests degrade immediately rather than hiding in an ever-growing queue.
- A bounded queue alone is incomplete: callers need retry guidance and the fallback must be safe for the product decision.
YOUR TURN Pause and predict before revealing the answer.
A cache key contains user_id and model_version but omits feature_snapshot. What failure can appear after a feature refresh?
Need a hint?
The same user and model may legitimately produce a different result from newer inputs.
Check your reasoning
The cache can return a prediction made from the old feature snapshot as if it were current. Include feature or data version and policy identity, or set a freshness rule that safely invalidates it.
Lessons 1, 2 in this course.
Connect the chapter to the course’s quantitative spine.
Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought. The queue utilization and little’s law below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Fast serving comes from bounded, version-aware work; graceful overload is a feature, not an afterthought. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Budget the request path” into a written invariant and a failing test.
- 02
Promote admission.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Fail before saturation” as the release checkpoint: Bound queues and retries, expose remaining capacity, and return a tested fallback or overload response while the system is still controllable.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.