Read the system through its scar tissue.
Each article separates what the primary source reports from our educational reconstruction. Diagnose the evidence before opening the reusable pattern.
Data & platform
5 investigations
Michelangelo: one ML platform across the lifecycle
How a shared platform joined data preparation, training, deployment, prediction, and monitoring for many teams.
role:platformlifecycle:trainingChronon: feature definitions across offline and online paths
A reconstruction of how one feature definition can serve historical training joins and fresh online values.
role:platformfailure:training-serving-skewA gigascale feature store with Redis
Serving fresh features at large scale while controlling key design, memory, availability, and operational load.
role:platformlifecycle:dataThe winding road through TFX and Kubeflow
What an organization learns when workflow technology meets real team boundaries and operating constraints.
role:platformlifecycle:trainingMetaflow: a human-centric path from notebook to production
How abstractions can preserve an approachable local workflow while adding versioning, scale, and production execution.
role:platformconcept:reproducibilityTraining & distribution
5 investigations
Horovod: make distributed training feel small
Ring all-reduce, small API surface, and the engineering required for multi-worker speedups.
concept:all-reducelifecycle:trainingScaling Kubernetes to 7,500 nodes
Control-plane, networking, and operational lessons from very large ML clusters.
role:platformlifecycle:trainingMegatron: composing parallelism for giant models
Choosing data, tensor, and pipeline parallelism around memory and communication limits.
lifecycle:trainingconcept:all-reducePerformant distributed checkpointing
Sharded, resumable model state without turning checkpoints into the training bottleneck.
tool:pytorchlifecycle:trainingCompiling production training models safely
Introducing compiler optimization while preserving numerical and operational correctness.
tool:pytorchlifecycle:trainingServing & cost
5 investigations
A platform for real-time predictions
Building shared online inference around contracts, reliability, and product-team needs.
lifecycle:inferencerole:platformGPU-accelerated inference at Pinterest
Finding the workload, batching, and runtime conditions under which accelerators improve production inference.
lifecycle:inferencetool:tritonRay batch inference: coordinating data and model work
Scaling offline inference by coordinating reads, transforms, accelerators, writes, and retry boundaries.
lifecycle:inferencerole:platformTail utilization in ads inference
Why fleet-average utilization can coexist with overloaded shards and poor tail latency.
lifecycle:inferencefailure:latencyBuilding a shared GenAI application stack
The layers around a language model: gateway, retrieval, orchestration, evaluation, safety, and observability.
track:nlp-llmconcept:ragDrift & incidents
5 investigations
D3: automated data-drift detection
Detecting changes across many features without turning every statistical difference into an incident.
failure:driftlifecycle:monitoringWhen a generated feature file caused a global outage
A production incident reconstructed as an ML dependency and failure-containment lesson.
failure:supply-chainrole:reliabilityMonitoring ML models for bot detection
Observing an adversarial classifier when labels are partial and the traffic distribution fights back.
lifecycle:monitoringfailure:driftRaising the bar on ML deployment safety
Release gates and progressive exposure for models whose regressions may be hard to see immediately.
lifecycle:inferencerole:reliabilityOff-policy correction for recommender feedback
Learning and evaluating from feedback collected by an older ranking policy.
track:recommendation-searchfailure:feedback-loopFairness, privacy & security
4 investigations
Auditing and retiring an image-cropping algorithm
A public audit that shows how product context, user control, and model bias interact.
concept:fairnessmodality:imageA fairness toolkit for large-scale AI
Making subgroup evaluation and mitigation reusable across many models and teams.
concept:fairnessrole:platformPrivate training for on-device language models
Balancing utility, device constraints, aggregation, and formal privacy protection.
concept:privacylifecycle:trainingStripe Radar: learning fraud under adversarial pressure
Production fraud detection as a global, delayed-label, cost-sensitive, adversarial decision system.
algorithm:xgboostconcept:fairness