Randomized experiments and guardrails
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree.
Model quality decays while every infrastructure dashboard stays green.
This chapter moves from write the causal question to read effect with guards. We will make the idea concrete in a small code example before placing it back inside a production system with release error budget and label maturity.
Report effect and uncertainty at the correct unit alongside every preregistered harm and system guardrail.
Verify the behavior illustrated in experiment.py, then measure decision utility, slice reliability, and system latency.
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
An offline gain says a model behaves differently on recorded data; it does not prove the product outcome will improve. Randomized experiments estimate causal impact by making treatment assignment independent of the potential outcomes—when assignment and analysis are implemented correctly.
Choose the randomization unit where interference and repeated observations are manageable: user, account, market, or time block. Analyze at that unit or use an estimator that accounts for clustering. Before estimating effects, verify assignment, exposure, sample-ratio balance, logging, and invariant pre-treatment metrics.
Pre-register a primary outcome, minimum detectable effect, duration, slices, and guardrails such as latency, complaints, safety, and diversity. Repeated peeking inflates false positives unless the design uses sequential methods. Statistical significance is not launch permission: effect size, uncertainty, operational cost, and guardrails all matter.
- SRM
- confidence intervals
- guardrails
- champion/challenger
Follow the reasoning before touching the code.
- 01
Write the causal question
Name population, intervention, control, randomization unit, exposure, outcome window, and decision the result will inform.
- 02
Validate before estimating
Check assignment ratios, sample-ratio mismatch, exposure consistency, missing outcomes, and pre-treatment balance.
- 03
Read effect with guards
Report effect and uncertainty at the correct unit alongside every preregistered harm and system guardrail.
import numpy as np
control = np.array([0, 1, 0, 0, 1, 0], dtype=float)
treatment = np.array([1, 1, 0, 1, 1, 0], dtype=float)
effect = treatment.mean() - control.mean()
se = np.sqrt(treatment.var(ddof=1) / len(treatment)
+ control.var(ddof=1) / len(control))
print('effect:', round(effect, 3))
print('approx 95% interval:', (effect - 1.96 * se, effect + 1.96 * se))Estimate a difference in means at the randomized unit
What to notice
- The point estimate is positive, but the tiny sample produces a wide interval and weak decision evidence.
- Each array element must represent an independent randomized unit; using sessions from the same users would understate uncertainty.
YOUR TURN Pause and predict before revealing the answer.
Users are randomized, but the analysis treats every one of their sessions as independent. What bias appears in the standard error?
Need a hint?
Sessions from one user tend to be correlated.
Check your reasoning
The effective sample size is overstated, usually making the standard error too small and the result look more certain than it is.
Lessons 1, 2, 3 in this course.
Connect the chapter to the course’s quantitative spine.
Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree. The difference-in-means experiment estimate below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Difference-in-means experiment estimate
| Symbol | Meaning / shape / unit |
|---|---|
Ȳ_T, Ȳ_C | treatment/control means |
s² | sample variance |
n | independent randomized units |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: Ȳ_T, Ȳ_C, s², n.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c19(inputs)
assert is_finite(value)Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Drift triage lab
Increase shift magnitude and compare statistical signal with estimated business impact.
Assumption: Ten-thousand observations; model sensitivity fixed at 0.18 utility points per σ.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| global | 0.60σ | 10.8% |
| new users | 0.67σ | 12.7% |
| mobile | 0.74σ | 14.7% |
| region A | 0.82σ | 16.6% |
| region B | 0.89σ | 18.6% |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Trust an experiment only after assignment, exposure, unit of analysis, uncertainty, and guardrails all agree. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Write the causal question” into a written invariant and a failing test.
- 02
Promote experiment.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Read effect with guards” as the release checkpoint: Report effect and uncertainty at the correct unit alongside every preregistered harm and system guardrail.
- 04
Serve or materialize behind explicit release error budget and label maturity.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
Daily peeking and session-level analysis of a user-randomized experiment produce false certainty while a regional regression hides. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Launch analysis and incident response
Analyze a synthetic launch, verify assignment, estimate effects, inspect slices, and choose continue, pause, or rollback.
- Experiment validity report
- Layered reliability diagnosis
- SLO and runbook
- Postmortem/retraining policy
Read primary material with a purpose.
Return to the opening failure.
Every release needs causal evidence, multi-layer observability, tested rollback, and a retraining policy that does not automate faults.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce joined data/model/service telemetry, drift policy, experiments, alerts, and runbooks.