Evaluation
Evaluation in the context of production machine-learning systems.
Lessons
Formal explanations, examples, simulations, and checkpoints.
Predict decisions, not just labels
A model score earns its place by improving a declared decision.
Problem framing, statistics & evaluationLabels, availability, and leakage
Train on what the system knew then, label with what became known later.
Problem framing, statistics & evaluationBaselines and honest splits
A baseline sets the floor; an honest split keeps the ruler straight.
Problem framing, statistics & evaluationMetrics, thresholds, and calibration
Measure ordering, repair probability meaning, then choose the policy the operation can actually run.
Problem framing, statistics & evaluationUncertainty and online experiments
Uncertainty tempers the estimate; randomization tests the cause; guardrails protect the product.
Problem framing, statistics & evaluationMetric selection by task and decision
Select metrics from the prediction target, decision, error costs, prevalence, slices, horizon, and deployment constraints—not from habit.
Unsupervised learning, anomalies & retrievalPCA and representation geometry
PCA preserves dominant geometric variation, not automatically the information your decision needs.
Unsupervised learning, anomalies & retrievalClustering as an operational hypothesis
A cluster becomes knowledge only when stable geometry connects to a repeatable human or system action.
Unsupervised learning, anomalies & retrievalDensity clusters and novel anomalies
Density methods flag what is locally disconnected; people and outcomes determine what is truly novel and important.
Unsupervised learning, anomalies & retrievalEmbeddings and similarity
Similarity is a modeling choice: the same vectors tell different stories under cosine and dot product.
Unsupervised learning, anomalies & retrievalApproximate retrieval under latency
Approximate retrieval is an explicit recall–latency contract, not a transparent substitute for exact search.
Courses & assignments
Dependency-authoritative learning units.
Problem framing, statistics & evaluation
A metric contract, leakage-safe validation, calibration analysis, and experiment plan.
Assignment: Late-delivery evaluation contractCourse 06Unsupervised learning, anomalies & retrieval
A PCA, clustering, anomaly, and nearest-neighbor retrieval investigation.
Assignment: Incident discovery and retrieval systemCasebook
Reported facts and course reconstructions.
Off-policy correction for recommender feedback
Feedback data identifies policy-conditioned behavior; correction needs overlap and controlled variance.
Twitter · Fairness, privacy & securityAuditing and retiring an image-cropping algorithm
Sometimes product redesign and user agency are stronger mitigations than model tuning.
LinkedIn · Fairness, privacy & securityA fairness toolkit for large-scale AI
Fairness infrastructure must be context-aware, governed, and integrated into decisions.