Shadow, canary, rollback, fallback, observability
A release is safe only when exposure is attributable, bounded, observable, and quickly reversible.
Launch traffic causes p99 spikes and leaves expensive GPUs idle.
This chapter moves from establish the baseline to practice reversal. We will make the idea concrete in a small code example before placing it back inside a production system with p99 under the product timeout.
Trigger rollback and fallback in a test environment, verify compatibility, and measure time to safe recovery.
Verify the behavior illustrated in rollout.yaml, then measure decision utility, slice reliability, and system latency.
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
A safe release is a sequence of evidence, not a single deployment event. Shadow traffic proves execution without influencing decisions; a canary exposes a controlled slice; rollback restores a known-good version; and fallback keeps the product usable when models or dependencies fail.
Shadowing mirrors representative requests to the candidate and compares outputs, errors, latency, and resource use without returning candidate decisions to users. A canary does affect a small, explicitly assigned population, so assignment, exposure logging, guardrails, and stop conditions must be correct before it starts.
Rollback should be rehearsed and include model, preprocessing, feature, schema, policy, and runtime compatibility. A fallback may be the previous model, a rule, cached result, or abstention; its acceptable behavior depends on harm. Observability must join request, artifact, feature, decision, and outcome identities so the team can identify exactly who saw what.
- canaries
- fallback
Follow the reasoning before touching the code.
- 01
Establish the baseline
Record current quality, latency, errors, cost, slices, and known failure behavior for the exact production version.
- 02
Expand exposure deliberately
Move from offline replay to shadow to canary to wider traffic only when predefined evidence and guardrails pass.
- 03
Practice reversal
Trigger rollback and fallback in a test environment, verify compatibility, and measure time to safe recovery.
candidate: risk-model-v42
baseline: risk-model-v41
stages:
- mode: shadow
traffic: 100%
- mode: canary
traffic: 2%
guards:
p99_ms: 120
error_rate: 0.005
slice_recall_drop: 0.02
on_guard_failure: rollback_and_use_rulesA rollout policy states progression and reversal before launch
What to notice
- Shadow can receive all mirrored traffic because it does not control the user decision, while the canary begins with small impact.
- The slice guard prevents an acceptable aggregate from hiding a concentrated regression.
YOUR TURN Pause and predict before revealing the answer.
The canary has better accuracy but feature-fetch errors double. Should it expand?
Need a hint?
A release gate covers the executable decision system, not only model quality.
Check your reasoning
No. Hold or rollback, diagnose the feature-path regression, and expand only when both decision quality and reliability guards pass.
Lessons 1, 2, 3, 4 in this course.
Connect the chapter to the course’s quantitative spine.
A release is safe only when exposure is attributable, bounded, observable, and quickly reversible. The queue utilization and little’s law below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Queue utilization and Little’s law
| Symbol | Meaning / shape / unit |
|---|---|
λ | arrival rate |
c | worker count |
μ | service rate per worker |
W | mean time in system |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: λ, c, μ, W.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c18(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Serving queue lab
Change concurrency and compare utilization, queue pressure, and tail latency.
Assumption: Four workers, 18 requests/s each, 850 ms timeout, burst factor 1.3.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| worker 1 | 10 active | healthy |
| worker 2 | 10 active | healthy |
| worker 3 | 10 active | healthy |
| worker 4 | 10 active | healthy |
| worker queue | 0 waiting | healthy |
Trace the complete operating path.
Carry the chapter’s central idea into the system: A release is safe only when exposure is attributable, bounded, observable, and quickly reversible. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Establish the baseline” into a written invariant and a failing test.
- 02
Promote rollout.yaml from illustration to a versioned implementation with representative fixtures.
- 03
Use “Practice reversal” as the release checkpoint: Trigger rollback and fallback in a test environment, verify compatibility, and measure time to safe recovery.
- 04
Serve or materialize behind explicit p99 under the product timeout.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Client retries amplify a slow dependency; readiness passes before warmup and newly scaled replicas worsen p99. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Production inference contract
Specify a typed service with warmup, batching, resilience, load testing, rollout, metrics, and fallback.
- API/artifact contract
- Capacity/load report
- Failure injection
- Shadow/canary/rollback design
Read primary material with a purpose.
Return to the opening failure.
Treat inference as a distributed system with explicit degradation, auditability, rollout, and reversal.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce batch and online paths with batching, autoscaling, load tests, canaries, and rollback.