concept:calibration

Calibration

Calibration in the context of production machine-learning systems.

Lessons

Formal explanations, examples, simulations, and checkpoints.

Use predict decisions, not just labels to move the evaluation production brief toward a defensible release.

Use labels, availability, and leakage to move the evaluation production brief toward a defensible release.

Use baselines and honest splits to move the evaluation production brief toward a defensible release.

Use metrics, thresholds, and calibration to move the evaluation production brief toward a defensible release.

Use uncertainty and online experiments to move the evaluation production brief toward a defensible release.

Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.

Use trees as learned decision rules to move the trees & boosting production brief toward a defensible release.

Use random forests and diversity to move the trees & boosting production brief toward a defensible release.

Use gradient boosting as error correction to move the trees & boosting production brief toward a defensible release.

Use xgboost: regularized boosting at scale to move the trees & boosting production brief toward a defensible release.

Use calibrating and explaining ensembles to move the trees & boosting production brief toward a defensible release.

Dependency-authoritative learning units.

A metric contract, leakage-safe validation, calibration analysis, and experiment plan.

Assignment: Late-delivery evaluation contract Course 05

Calibrated random-forest and XGBoost baselines with cost-sensitive thresholds.

Assignment: Rare-fraud ensemble decision system

Reported facts and course reconstructions.