Metric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
Teams report excellent accuracy, RMSE, mAP, or perplexity while the fraud queue, forecast, defect review, and assistant outcomes get worse.
This chapter moves from start from the harm to write the counterexample. We will make the idea concrete in a small code example before placing it back inside a production system with the decision latency, review capacity, horizon, and cost envelope for the task.
Choose the smallest metric set that measures model discrimination, probability/uncertainty quality, decision utility, and system constraints.
A versioned metric contract with direction, unit, aggregation, slices, uncertainty, threshold, guardrails, and stop conditions.
A team chooses ROC-AUC for rare-event review, MAPE with near-zero targets, pixel accuracy for tiny defects, and perplexity as the only assistant launch metric. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
There is no universally honest metric. A metric is honest only about the particular target, action, error tradeoff, and aggregation rule it was chosen to represent.
Accuracy can celebrate a classifier that ignores every rare positive. ROC-AUC can hide an unusable review queue, MAPE can explode near zero demand, and pixel accuracy can ignore a tiny defect mask. Each number has a question it answers and cases where that question is irrelevant.
A metric contract names a primary decision-facing measure, complementary diagnostics, slice and horizon aggregation, uncertainty, and system guardrails. Threshold-free quality and the chosen operating policy should remain separately visible.
- metric contracts
- proper scoring rules
- prevalence sensitivity
- ranking metrics
- forecast scale
- slice reliability
- decision utility
Follow the reasoning before touching the code.
- 01
Start from the harm
Describe what a false positive, false negative, large residual, or misplaced rank causes for a person or operation.
- 02
Choose the smallest portfolio
Pair one primary task metric with calibration or interval quality, critical slices, and latency or capacity guardrails.
- 03
Write the counterexample
Construct a plausible model that improves the chosen metric while worsening the decision; add a diagnostic that exposes it.
labels = [0] * 18 + [1, 1]
all_negative = [0] * 20
triage_model = [0] * 17 + [1, 1, 0]
def report(predictions):
tp = sum(p == 1 and y == 1 for p, y in zip(predictions, labels))
fp = sum(p == 1 and y == 0 for p, y in zip(predictions, labels))
fn = sum(p == 0 and y == 1 for p, y in zip(predictions, labels))
accuracy = sum(p == y for p, y in zip(predictions, labels)) / len(labels)
precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn)
return {"accuracy": accuracy, "precision": precision, "recall": recall}
print("all negative", report(all_negative))
print("triage", report(triage_model))Show why accuracy misses rare-event decision quality.
What to notice
- The all-negative classifier reaches 90% accuracy while recovering none of the positives.
- The triage model has the same 90% accuracy yet creates a queue with measurable precision and recall.
YOUR TURN Pause and predict before revealing the answer.
If reviewers can inspect only one case, which additional evaluation quantity should the contract emphasize?
Need a hint?
The operation cares about the very top of the ranked list, not every possible threshold equally.
Check your reasoning
Use precision@1 or expected utility at capacity one, accompanied by uncertainty and the relevant slice guardrails.
Lessons 1, 2, 3, 4, 5 in this course.
Connect the chapter to the course’s quantitative spine.
Choose metrics from the decision’s failure modes, not from the model family’s default report. The cost-weighted harmonic mean of precision and recall below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Cost-weighted harmonic mean of precision and recall
| Symbol | Meaning / shape / unit |
|---|---|
β | relative weight placed on recall |
precision | fraction of positive decisions that are correct |
recall | fraction of actual positives recovered |
F_β | one summary at a declared threshold |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: β, precision, recall, F_β.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c02(inputs)
assert is_finite(value)Choose from the task contract, not a favorite default.
Start from the decision, harm, target semantics, prevalence, aggregation unit, horizon, and operating constraint. Keep one primary metric, complementary diagnostics, and production guardrails.
| Task | Primary starting point | Pair with | Production guardrails | Never use alone |
|---|---|---|---|---|
| Binary probability | Log loss or Brier score; calibration error | PR-AUC for rare positives; ROC-AUC as discrimination context | Precision, recall, utility, and queue volume at the operating threshold | Accuracy or AUC alone |
| Hard classification | Balanced accuracy, MCC, or justified Fβ | Per-class confusion matrix; macro and micro views | Cost by slice, abstention, latency, and fallback outcomes | One aggregate accuracy |
| Multiclass / multilabel | Macro and per-label precision/recall/F1 | Micro metrics and top-k only when product semantics fit | Coverage, calibration, rare-label behavior, and policy capacity | Micro average alone |
| Regression | MAE for linear error cost; RMSE for large-error sensitivity | R² only against a meaningful baseline; quantile error when asymmetric | Residual slices, bias, uncertainty, and action cost | A metric chosen without units |
| Forecasting | MASE or scale-aware error by horizon; bias | Pinball loss and interval coverage/width | Inventory or staffing cost, horizon slices, and drift | MAPE near zero or one pooled horizon |
| Ranking / search | NDCG@k for graded order; MRR for first-hit tasks | Recall@k for candidate generation; MAP when relevant-set semantics fit | Coverage, diversity, freshness, position slices, latency, and online utility | Offline clicks treated as unbiased relevance |
| Anomaly detection | Precision or analyst yield at alert budget | Event recall, false alerts per time unit, detection delay | Incident cost, alert fatigue, novelty slices, and investigation latency | ROC-AUC on extreme imbalance |
| Detection | mAP across IoU thresholds, classes, sizes, and slices | Recall at an operating confidence threshold | Calibration, duplicate rate, review load, and device latency | One mAP number |
| Segmentation | Per-class IoU or Dice plus boundary quality | Pixel-level precision/recall and calibration where useful | Small-object slices, topology failures, latency, and correction effort | Pixel accuracy on background-heavy masks |
| Clustering | Stability and downstream/operator utility | ARI or NMI only when trustworthy labels exist; silhouette as a diagnostic | Coverage, actionability, drift, and manual validation | Internal geometry declared product success |
| Language generation | Task success, groundedness, factuality, and safety | Token loss/perplexity as an intrinsic diagnostic; calibrated judge or human preference with audit | Citation validity, refusal behavior, latency, cost, and adversarial slices | Perplexity or one opaque judge score |
Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Cost-aware threshold
Move the cutoff over a fixed set of scored transactions.
Assumption: Fixed 12-event dataset; false positive costs 2 units and false negative costs 12.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| 0.92 | fraud | review |
| 0.84 | fraud | review |
| 0.76 | legit | review |
| 0.68 | fraud | review |
| 0.59 | legit | review |
| 0.51 | fraud | review |
| 0.43 | legit | pass |
| 0.37 | legit | pass |
| 0.31 | fraud | pass |
| 0.22 | legit | pass |
| 0.14 | legit | pass |
| 0.08 | legit | pass |
Trace the complete operating path.
Carry the chapter’s central idea into the system: Choose metrics from the decision’s failure modes, not from the model family’s default report. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Start from the harm” into a written invariant and a failing test.
- 02
Promote rare_event_metrics.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Write the counterexample” as the release checkpoint: Construct a plausible model that improves the chosen metric while worsening the decision; add a diagnostic that exposes it.
- 04
Serve or materialize behind explicit the decision latency, review capacity, horizon, and cost envelope for the task.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
A team chooses ROC-AUC for rare-event review, MAPE with near-zero targets, pixel accuracy for tiny defects, and perplexity as the only assistant launch metric. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Metric-contract portfolio
Write complete selection contracts for rare fraud, ETA regression, search ranking, and visual defect detection.
- Primary metric and decision rationale for four tasks
- Secondary diagnostics, thresholds, and guardrails
- Aggregation, horizon, slice, and uncertainty definitions
- Counterexample where each selected metric misleads
- Release and rollback decision template
Staged hints
Failure diagnosis
- If offline numbers look impossible, audit prediction-time availability before model code.
- If quality is sound but the contract fails, measure the exact serialized and served path.
- If one slice regresses, preserve that slice as a permanent release gate.
Record an honest attempt to unlock the reference solution.
Read primary material with a purpose.
scikit-learn model evaluation guide
45 min · last checked 26 Aug 2026
↗RECOMMENDED · Official docsGoogle ML Crash Course — classification metrics
25 min · last checked 26 Aug 2026
↗RECOMMENDED · EngineeringForecast evaluation for data scientists
30 min · last checked 26 Aug 2026
↗DEEP DIVE · INDEXFind a primary implementation or benchmark
30–90 min · verify task, dataset, and evaluation match
↗Return to the opening failure.
Every model card names why each metric matches the action, which aggregation and horizon it uses, required slice/uncertainty views, and the threshold or policy that turns scores into outcomes.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a metric contract, leakage-safe validation, calibration analysis, and experiment plan.