Operators, product owners, and people affected by the evaluation decision.
Foundations · weeks 2–3
Problem framing, statistics & evaluation
An ETA model improves RMSE while increasing missed-delivery cost.
Opening production brief
A measurable problem, before a model.
decision utility, prediction cutoff, censoring, temporal splits with version, owner, event time, and availability contracts.
An ETA model improves RMSE while increasing missed-delivery cost. Every solution must state latency, cost, capacity, and fallback limits.
Offline evidence plus a deployable resolution: A metric contract, leakage-safe validation, calibration analysis, and experiment plan.
Frame the decision
Turn the product request into a decision and label contract.
Validate honestly
Build leakage-safe baselines and validation.
- 03↗
Baselines and honest splits
A baseline sets the floor; an honest split keeps the ruler straight.
65–90 min · checkpoint · deterministic lab - 04↗
Metrics, thresholds, and calibration
Measure ordering, repair probability meaning, then choose the policy the operation can actually run.
65–90 min · checkpoint · deterministic lab
Choose metrics, objectives, and policy
Select metrics, optimization losses, thresholds, uncertainty, and online evidence from the task.
- 05↗
Uncertainty and online experiments
Uncertainty tempers the estimate; randomization tests the cause; guardrails protect the product.
65–90 min · checkpoint · deterministic lab - 06↗
Metric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
65–90 min · checkpoint · deterministic lab - HW↗
Late-delivery evaluation contract
Specify a cutoff-safe target, validation design, cost/capacity threshold, calibration report, uncertainty, and controlled experiment.
Rubric · staged hints · reference resolution
Prerequisite guidance
1 concepts make this faster.
Prerequisites are guidance, never hard gates.