Operators, product owners, and people affected by the evaluation decision.
Foundations · weeks 2–3
Problem framing, statistics & evaluation
An ETA model improves RMSE while increasing missed-delivery cost.
Opening production brief
A measurable problem, before a model.
decision utility, prediction cutoff, censoring, temporal splits with version, owner, event time, and availability contracts.
An ETA model improves RMSE while increasing missed-delivery cost. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: A metric contract, leakage-safe validation, calibration analysis, and experiment plan.
Frame the decision
Turn the product request into a decision and label contract.
- 01↗
Predict decisions, not just labels
Use predict decisions, not just labels to move the evaluation production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 02↗
Labels, availability, and leakage
Use labels, availability, and leakage to move the evaluation production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Validate honestly
Build leakage-safe baselines and validation.
- 03↗
Baselines and honest splits
Use baselines and honest splits to move the evaluation production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 04↗
Metrics, thresholds, and calibration
Use metrics, thresholds, and calibration to move the evaluation production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab
Choose metrics, objectives, and policy
Select metrics, optimization losses, thresholds, uncertainty, and online evidence from the task.
- 05↗
Uncertainty and online experiments
Use uncertainty and online experiments to move the evaluation production brief toward a defensible release.
65–90 min · checkpoint · deterministic lab - 06↗
Metric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
65–90 min · checkpoint · deterministic lab - HW↗
Late-delivery evaluation contract
Specify a cutoff-safe target, validation design, cost/capacity threshold, calibration report, uncertainty, and controlled experiment.
Rubric · staged hints · reference resolution
Prerequisite guidance
1 concepts make this faster.
Prerequisites are guidance, never hard gates.