Evaluation
Evaluation in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Predict decisions, not just labels
Use predict decisions, not just labels to move the evaluation production brief toward a defensible release.
Problem framing, statistics & evaluationLabels, availability, and leakage
Use labels, availability, and leakage to move the evaluation production brief toward a defensible release.
Problem framing, statistics & evaluationBaselines and honest splits
Use baselines and honest splits to move the evaluation production brief toward a defensible release.
Problem framing, statistics & evaluationMetrics, thresholds, and calibration
Use metrics, thresholds, and calibration to move the evaluation production brief toward a defensible release.
Problem framing, statistics & evaluationUncertainty and online experiments
Use uncertainty and online experiments to move the evaluation production brief toward a defensible release.
Problem framing, statistics & evaluationMetric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
Unsupervised learning, anomalies & retrievalPCA and representation geometry
Use pca and representation geometry to move the unsupervised & retrieval production brief toward a defensible release.
Unsupervised learning, anomalies & retrievalClustering as an operational hypothesis
Use clustering as an operational hypothesis to move the unsupervised & retrieval production brief toward a defensible release.
Unsupervised learning, anomalies & retrievalDensity clusters and novel anomalies
Use density clusters and novel anomalies to move the unsupervised & retrieval production brief toward a defensible release.
Unsupervised learning, anomalies & retrievalEmbeddings and similarity
Use embeddings and similarity to move the unsupervised & retrieval production brief toward a defensible release.
Unsupervised learning, anomalies & retrievalApproximate retrieval under latency
Use approximate retrieval under latency to move the unsupervised & retrieval production brief toward a defensible release.
Courses & assignments
Dependency-authoritative learning units.
Problem framing, statistics & evaluation
A metric contract, leakage-safe validation, calibration analysis, and experiment plan.
Assignment: Late-delivery evaluation contractCourse 06Unsupervised learning, anomalies & retrieval
A PCA, clustering, anomaly, and nearest-neighbor retrieval investigation.
Assignment: Incident discovery and retrieval systemCasebook
Reported facts and course reconstructions.
Off-policy correction for recommender feedback
Feedback data identifies policy-conditioned behavior; correction needs overlap and controlled variance.
Twitter · Fairness, privacy & securityAuditing and retiring an image-cropping algorithm
Sometimes product redesign and user agency are stronger mitigations than model tuning.
LinkedIn · Fairness, privacy & securityA fairness toolkit for large-scale AI
Fairness infrastructure must be context-aware, governed, and integrated into decisions.