Choose batch, online, streaming, edge, or hybrid
Place inference where the decision’s deadline, freshness, and failure semantics can actually be met.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This chapter moves from start with the action deadline to choose and reconcile paths. We will make the idea concrete in a small code example before placing it back inside a production system with p99 under the product timeout.
Select the simplest mode or hybrid, then define shared artifact, preprocessing, version, and correction contracts.
Verify the behavior illustrated in serving_mode.py, then measure decision utility, slice reliability, and system latency.
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
Serving mode follows the decision clock. A monthly segmentation job, a checkout risk score, a millisecond sensor alarm, and an intermittently connected device need different paths even if they use the same model family.
Batch inference favors throughput and reproducibility when decisions can wait. Online inference answers one request under a latency SLO. Streaming reacts continuously to events and must define event-time and replay semantics. Edge inference moves computation near the input for latency, privacy, or connectivity, but makes updates and observability harder.
Hybrid systems are common: batch precomputes embeddings, online logic scores a small candidate set, streaming refreshes counters, and an edge model provides a local fallback. Choose the smallest combination that meets freshness, availability, privacy, cost, and correction requirements, then state how the paths remain semantically consistent.
- prediction contracts
- REST/gRPC
- batching
- timeouts
Follow the reasoning before touching the code.
- 01
Start with the action deadline
Write when the prediction must exist to change a decision and how stale it may be before it loses value.
- 02
Add operating constraints
Account for arrival pattern, connectivity, data locality, privacy, throughput, update cadence, and failure behavior.
- 03
Choose and reconcile paths
Select the simplest mode or hybrid, then define shared artifact, preprocessing, version, and correction contracts.
def serving_mode(deadline_ms, offline_ok, continuous_events, disconnected):
if disconnected:
return 'edge or edge + sync'
if offline_ok:
return 'batch'
if continuous_events:
return 'streaming + online state'
if deadline_ms < 1000:
return 'online'
return 'micro-batch'
print(serving_mode(80, False, False, False))Make the serving decision from explicit constraints
What to notice
- The function is a discussion aid, not an architecture generator; real constraints can point to more than one path.
- Latency alone does not imply edge serving—connectivity, privacy, update policy, and device capacity must support it.
YOUR TURN Pause and predict before revealing the answer.
Recommendations may be refreshed hourly, but price and availability must be current at click time. What serving shape fits?
Need a hint?
Separate expensive candidate computation from volatile decision-time facts.
Check your reasoning
Use batch or streaming to precompute candidates, then an online step to filter or rescore with current price and availability—a hybrid path.
GPU training, Efficient inference, ML platforms
Connect the chapter to the course’s quantitative spine.
Place inference where the decision’s deadline, freshness, and failure semantics can actually be met. The queue utilization and little’s law below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Place inference where the decision’s deadline, freshness, and failure semantics can actually be met. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Start with the action deadline” into a written invariant and a failing test.
- 02
Promote serving_mode.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Choose and reconcile paths” as the release checkpoint: Select the simplest mode or hybrid, then define shared artifact, preprocessing, version, and correction contracts.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.