Package model, preprocessing, schema, and runtime
The deployable artifact is the complete raw-input-to-output program, not the weights alone.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This chapter moves from define raw-to-decision scope to replay golden requests. We will make the idea concrete in a small code example before placing it back inside a production system with p99 under the product timeout.
Load the packaged artifact in a fresh process and verify raw-input outputs, errors, defaults, and compatibility before promotion.
Verify the behavior illustrated in artifact_manifest.py, then measure decision utility, slice reliability, and system latency.
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A model file is only one piece of an executable prediction. The runtime must also know how to validate raw input, reproduce transforms, map outputs, load compatible dependencies, and identify the exact bundle being served.
Training-serving skew appears when production rebuilds preprocessing from memory or independently maintained code. Package fitted encoders, vocabulary, normalization state, feature order, schema, model weights, and postprocessing behind one tested prediction interface. Prefer safe, portable formats when possible and treat arbitrary serialized code as executable supply-chain input.
The bundle needs a signature: accepted fields, types, ranges, defaults, output semantics, and version. A golden parity test should begin with a raw request, cross serialization and process boundaries, and compare the final output with a stored expectation under a documented tolerance.
- batching
- timeouts
- circuit breakers
- queueing
Follow the reasoning before touching the code.
- 01
Define raw-to-decision scope
List every stateful and stateless operation from external request through validation, features, model, calibration, and policy.
- 02
Build one immutable bundle
Pin model, preprocessing state, schema, runtime, dependencies, and checksums under one release identity.
- 03
Replay golden requests
Load the packaged artifact in a fresh process and verify raw-input outputs, errors, defaults, and compatibility before promotion.
from hashlib import sha256
from pathlib import Path
expected = {
'model.onnx': '9f5d...',
'schema.json': '2aa1...',
'transform.json': '71c0...',
}
for name, digest in expected.items():
actual = sha256(Path(name).read_bytes()).hexdigest()
if not actual.startswith(digest.removesuffix('...')):
raise RuntimeError(f'checksum mismatch: {name}')Verify every file in a prediction bundle before loading it
What to notice
- Verification happens before deserialization, so a mismatched file cannot silently enter the prediction path.
- Checksums prove byte identity, not trust by themselves; the manifest still needs an authenticated source and approval chain.
YOUR TURN Pause and predict before revealing the answer.
The model checksum matches, but transform.json changed. Can the release proceed if an offline metric still looks good?
Need a hint?
Think about feature order, learned state, parity, and the identity of the evaluated system.
Check your reasoning
No. The executable system changed. Recompute the bundle identity, rerun golden parity and evaluation, and promote only the exact model-transform-schema-runtime combination that passed.
Lessons 1 in this course.
Connect the chapter to the course’s quantitative spine.
The deployable artifact is the complete raw-input-to-output program, not the weights alone. The queue utilization and little’s law below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
Carry the chapter’s central idea into the system: The deployable artifact is the complete raw-input-to-output program, not the weights alone. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Define raw-to-decision scope” into a written invariant and a failing test.
- 02
Promote artifact_manifest.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Replay golden requests” as the release checkpoint: Load the packaged artifact in a fresh process and verify raw-input outputs, errors, defaults, and compatibility before promotion.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.