Choose batch, online, streaming, edge, or hybrid
Use choose batch, online, streaming, edge, or hybrid to move the serving systems production brief toward a defensible release.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This lesson isolates choose batch, online, streaming, edge, or hybrid as one decision inside that system. The people affected are product users and operators; the learning data must carry event time, availability time, ownership, and version; and the operating envelope is p99 under the product timeout.
Choose whether and how to use prediction contracts at a declared prediction cutoff.
Measure decision utility alongside calibration, slice reliability, and system latency—not model score alone.
Launch traffic causes p99 spikes and leaves expensive GPUs idle. An unsafe release must degrade to a named baseline or the last known-good version.
Build the mental model before the machinery.
The core move is to treat choose batch, online, streaming, edge, or hybrid as a contract between data, a computation, and an action. Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal. The implementation becomes easier to debug once you can state which inputs exist, which state is learned, what output means, and what must remain invariant after serialization.
prediction contracts
Define it in a hand-checkable form and name the prediction-time inputs.
REST/gRPC
Connect it to the production metric and identify what it cannot guarantee.
batching
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
timeouts
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
GPU training, Efficient inference, ML platforms
Name every symbol. Check every shape.
Queue utilization and Little’s law is the central invariant for this lesson. The formula is useful only when its inputs match the production cutoff and its output maps to an action.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Calculate it small. Shape it realistically. Break it on purpose.
A result you can reproduce on paper
At 80 requests/s, two 50 requests/s workers have ρ=.8 yet bursts can still form a long queue.
- Write every input and unit.
- Substitute values into the queue utilization and little’s law equation above.
- Compare the result to one simple baseline and explain the direction of the difference.
The same reasoning under real constraints
A typed payment-risk API fetches versioned features, scores XGBoost, logs lineage, and uses a rules fallback.
The production record includes the data snapshot, transformation state, artifact identity, cutoff, score, decision, and the version of the policy that consumed it.
The attractive result you should reject
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99.
Diagnostic: replay the smallest failing slice from immutable inputs, then compare each boundary rather than retuning the model.
Change one assumption and make the tradeoff visible.
This lab runs predefined TypeScript only. It never executes learner code. Use the slider, numeric input, reset, live text, or table—the computation is the same.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
- 01
Validate and version prediction contracts.
- 02
Compute choose batch, online, streaming, edge, or hybrid from prediction-time-safe inputs.
- 03
Persist model, feature, and configuration identities together.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.