Metric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
Teams report excellent accuracy, RMSE, mAP, or perplexity while the fraud queue, forecast, defect review, and assistant outcomes get worse.
This lesson isolates metric selection by task and decision as one decision inside that system. The people affected are product users and operators; the learning data must carry event time, availability time, ownership, and version; and the operating envelope is the decision latency, review capacity, horizon, and cost envelope for the task.
Choose the smallest metric set that measures model discrimination, probability/uncertainty quality, decision utility, and system constraints.
A versioned metric contract with direction, unit, aggregation, slices, uncertainty, threshold, guardrails, and stop conditions.
Teams report excellent accuracy, RMSE, mAP, or perplexity while the fraud queue, forecast, defect review, and assistant outcomes get worse. An unsafe release must degrade to a named baseline or the last known-good version.
Build the mental model before the machinery.
The core move is to treat metric selection by task and decision as a contract between data, a computation, and an action. Every model card names why each metric matches the action, which aggregation and horizon it uses, required slice/uncertainty views, and the threshold or policy that turns scores into outcomes. The implementation becomes easier to debug once you can state which inputs exist, which state is learned, what output means, and what must remain invariant after serialization.
metric contracts
Define it in a hand-checkable form and name the prediction-time inputs.
proper scoring rules
Connect it to the production metric and identify what it cannot guarantee.
prevalence sensitivity
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
ranking metrics
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
forecast scale
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
slice reliability
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
decision utility
Stress it with a slice, a temporal boundary, and a failure-safe alternative.
Lessons 1, 2, 3, 4, 5 in this course.
Name every symbol. Check every shape.
Cost-weighted harmonic mean of precision and recall is the central invariant for this lesson. The formula is useful only when its inputs match the production cutoff and its output maps to an action.
Cost-weighted harmonic mean of precision and recall
| Symbol | Meaning / shape / unit |
|---|---|
β | relative weight placed on recall |
precision | fraction of positive decisions that are correct |
recall | fraction of actual positives recovered |
F_β | one summary at a declared threshold |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: β, precision, recall, F_β.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c02(inputs)
assert is_finite(value)Choose from the task contract, not a favorite default.
Start from the decision, harm, target semantics, prevalence, aggregation unit, horizon, and operating constraint. Keep one primary metric, complementary diagnostics, and production guardrails.
| Task | Primary starting point | Pair with | Production guardrails | Never use alone |
|---|---|---|---|---|
| Binary probability | Log loss or Brier score; calibration error | PR-AUC for rare positives; ROC-AUC as discrimination context | Precision, recall, utility, and queue volume at the operating threshold | Accuracy or AUC alone |
| Hard classification | Balanced accuracy, MCC, or justified Fβ | Per-class confusion matrix; macro and micro views | Cost by slice, abstention, latency, and fallback outcomes | One aggregate accuracy |
| Multiclass / multilabel | Macro and per-label precision/recall/F1 | Micro metrics and top-k only when product semantics fit | Coverage, calibration, rare-label behavior, and policy capacity | Micro average alone |
| Regression | MAE for linear error cost; RMSE for large-error sensitivity | R² only against a meaningful baseline; quantile error when asymmetric | Residual slices, bias, uncertainty, and action cost | A metric chosen without units |
| Forecasting | MASE or scale-aware error by horizon; bias | Pinball loss and interval coverage/width | Inventory or staffing cost, horizon slices, and drift | MAPE near zero or one pooled horizon |
| Ranking / search | NDCG@k for graded order; MRR for first-hit tasks | Recall@k for candidate generation; MAP when relevant-set semantics fit | Coverage, diversity, freshness, position slices, latency, and online utility | Offline clicks treated as unbiased relevance |
| Anomaly detection | Precision or analyst yield at alert budget | Event recall, false alerts per time unit, detection delay | Incident cost, alert fatigue, novelty slices, and investigation latency | ROC-AUC on extreme imbalance |
| Detection | mAP across IoU thresholds, classes, sizes, and slices | Recall at an operating confidence threshold | Calibration, duplicate rate, review load, and device latency | One mAP number |
| Segmentation | Per-class IoU or Dice plus boundary quality | Pixel-level precision/recall and calibration where useful | Small-object slices, topology failures, latency, and correction effort | Pixel accuracy on background-heavy masks |
| Clustering | Stability and downstream/operator utility | ARI or NMI only when trustworthy labels exist; silhouette as a diagnostic | Coverage, actionability, drift, and manual validation | Internal geometry declared product success |
| Language generation | Task success, groundedness, factuality, and safety | Token loss/perplexity as an intrinsic diagnostic; calibrated judge or human preference with audit | Citation validity, refusal behavior, latency, cost, and adversarial slices | Perplexity or one opaque judge score |
Calculate it small. Shape it realistically. Break it on purpose.
A result you can reproduce on paper
Two classifiers are 95% accurate on a 5% positive dataset. One predicts every row negative; the other recovers 70% of positives at 50% precision. Compare accuracy, PR behavior, and action cost.
- Write every input and unit.
- Substitute values into the cost-weighted harmonic mean of precision and recall equation above.
- Compare the result to one simple baseline and explain the direction of the difference.
The same reasoning under real constraints
Create a metric contract for fraud, demand forecasting, retrieval, segmentation, and language generation that names a primary decision metric, calibration or uncertainty measure, slice views, and system guardrails.
The production record includes the data snapshot, transformation state, artifact identity, cutoff, score, decision, and the version of the policy that consumed it.
The attractive result you should reject
A team chooses ROC-AUC for rare-event review, MAPE with near-zero targets, pixel accuracy for tiny defects, and perplexity as the only assistant launch metric.
Diagnostic: replay the smallest failing slice from immutable inputs, then compare each boundary rather than retuning the model.
Change one assumption and make the tradeoff visible.
This lab runs predefined TypeScript only. It never executes learner code. Use the slider, numeric input, reset, live text, or table—the computation is the same.
Cost-aware threshold
Move the cutoff over a fixed set of scored transactions.
Assumption: Fixed 12-event dataset; false positive costs 2 units and false negative costs 12.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| 0.92 | fraud | review |
| 0.84 | fraud | review |
| 0.76 | legit | review |
| 0.68 | fraud | review |
| 0.59 | legit | review |
| 0.51 | fraud | review |
| 0.43 | legit | pass |
| 0.37 | legit | pass |
| 0.31 | fraud | pass |
| 0.22 | legit | pass |
| 0.14 | legit | pass |
| 0.08 | legit | pass |
Trace the complete operating path.
- 01
Validate and version metric contracts.
- 02
Compute metric selection by task and decision from prediction-time-safe inputs.
- 03
Persist model, feature, and configuration identities together.
- 04
Serve or materialize behind explicit the decision latency, review capacity, horizon, and cost envelope for the task.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A team chooses ROC-AUC for rare-event review, MAPE with near-zero targets, pixel accuracy for tiny defects, and perplexity as the only assistant launch metric. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Metric-contract portfolio
Write complete selection contracts for rare fraud, ETA regression, search ranking, and visual defect detection.
- Primary metric and decision rationale for four tasks
- Secondary diagnostics, thresholds, and guardrails
- Aggregation, horizon, slice, and uncertainty definitions
- Counterexample where each selected metric misleads
- Release and rollback decision template
Staged hints
Failure diagnosis
- If offline numbers look impossible, audit prediction-time availability before model code.
- If quality is sound but the contract fails, measure the exact serialized and served path.
- If one slice regresses, preserve that slice as a permanent release gate.
Record an honest attempt to unlock the reference solution.
Read primary material with a purpose.
scikit-learn model evaluation guide
45 min · last checked 26 Aug 2026
↗RECOMMENDED · Official docsGoogle ML Crash Course — classification metrics
25 min · last checked 26 Aug 2026
↗RECOMMENDED · EngineeringForecast evaluation for data scientists
30 min · last checked 26 Aug 2026
↗DEEP DIVE · INDEXFind a primary implementation or benchmark
30–90 min · verify task, dataset, and evaluation match
↗Return to the opening failure.
Every model card names why each metric matches the action, which aggregation and horizon it uses, required slice/uncertainty views, and the threshold or policy that turns scores into outcomes.
For this lesson, the release evidence is a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a metric contract, leakage-safe validation, calibration analysis, and experiment plan.