Loss functions by task, data, and deployment constraints
Choose and debug training objectives as optimization surrogates for the real task, including imbalance, outliers, uncertainty, ranking, dense prediction, and generation.
Training loss falls steadily, yet rare-event precision, segmentation quality, and forecast decisions regress after deployment.
This chapter moves from write the tensor contract to evaluate the actual decision. We will make the idea concrete in a small code example before placing it back inside a production system with stable gradients, representative batch composition, and compute that fits the training window.
Choose a differentiable objective whose inductive bias and gradient behavior fit the target, noise, imbalance, and output contract.
Training stability and held-out decision metrics, calibration/uncertainty, slice behavior, and resource use—not loss value alone.
The team uses MSE for every target, applies softmax before a numerically stable logits loss, weights rare examples twice, and assumes a lower training loss guarantees a better product decision. An unsafe release must degrade to a named baseline or the last known-good version.
Build the idea one layer at a time.
The loss is the signal the optimizer can hear, not the full definition of product success. It must be differentiable and stable while emphasizing the errors, uncertainty, and output structure that matter for the task.
MSE gives large residuals quadratically large influence; MAE treats residual magnitude linearly; Huber is quadratic near zero and linear in its tails. Classification, ranking, segmentation, and probabilistic forecasting require different output and reduction contracts. Class weights, sampling, and focal terms all alter gradient emphasis and can accidentally double-correct imbalance.
Select a loss from target encoding, noise, asymmetry, calibration needs, and numerical constraints, then gate release with independent decision metrics. Log unreduced losses and gradient norms by slice: one pleasant average can conceal a rare class receiving nearly all the update or a mask excluding every positive pixel.
- empirical risk
- surrogate objectives
- proper losses
- robust regression
- class weighting
- focal loss
- contrastive learning
- multi-objective training
Follow the reasoning before touching the code.
- 01
Write the tensor contract
State target values, model-output shape, logits versus probabilities, masks, per-example formula, and reduction before calling a library function.
- 02
Inspect gradient influence
Use representative easy, hard, noisy, rare, and masked examples to see which observations dominate parameter updates under each candidate loss.
- 03
Evaluate the actual decision
Hold the split and model budget fixed while comparing losses on calibration, thresholded utility, slices, stability, and compute—not training loss magnitude across unlike objectives.
import numpy as np
residuals = np.array([0.2, -0.5, 0.8, 10.0])
delta = 1.0
absolute = np.abs(residuals)
mse = 0.5 * residuals ** 2
mae = absolute
huber = np.where(absolute <= delta,
0.5 * residuals ** 2,
delta * (absolute - 0.5 * delta))
for name, values in [('MSE', mse), ('MAE', mae), ('Huber', huber)]:
print(name, values.round(2), 'mean', round(values.mean(), 2))Compare how regression losses treat one extreme residual.
What to notice
- The residual of 10 contributes 50 under half-squared error but only 9.5 under Huber with delta one.
- A smaller scalar loss is not inherently better across different formulas; compare fitted models on a fixed held-out decision metric.
YOUR TURN Pause and predict before revealing the answer.
What happens to Huber when delta becomes extremely large relative to every residual?
Need a hint?
Check which branch of the piecewise definition every example uses.
Check your reasoning
Every residual falls in the quadratic branch, so Huber behaves like half-squared error and loses its robust-tail advantage.
Lessons 1, 2, 3, 4, 5, 6 in this course.
Connect the chapter to the course’s quantitative spine.
A loss chooses which mistakes drive learning; an evaluation contract decides whether that learning is useful. The regularized empirical risk below is the shared course-level lens that helps quantify the surrounding system. Name each symbol, check its units or shape, and use only values that were available at the production cutoff.
Regularized empirical risk
| Symbol | Meaning / shape / unit |
|---|---|
θ | trainable parameters |
L | per-example surrogate loss |
f_θ(x_i) | model output with declared shape and semantics |
Ω | regularizer |
λ | regularization strength |
Open derivation and numerical substitution
Start from the production quantity being optimized, substitute the observed values with their declared units, then isolate the model-controlled term. Preserve shape annotations at each step so broadcasting or aggregation cannot silently change the result.
- Write the named inputs: θ, L, f_θ(x_i), Ω, λ.
- Substitute one small, hand-checkable batch before vectorizing.
- Calculate an independent reference value and compare within a declared tolerance.
# equation → code contract
inputs = validate_shapes_and_units(batch)
value = compute_c09(inputs)
assert is_finite(value)Choose from the task contract, not a favorite default.
Declare target and output shapes, logits versus probabilities, reduction, weights, masks, numerical stability, and gradient behavior before comparing objectives.
| Task | Start with | Change when | Common failure |
|---|---|---|---|
| Binary classification | One raw logit + BCE with logits | Class weighting or focal loss after measuring imbalance, noise, and calibration | Double sigmoid; weights interpreted as calibrated probability |
| Exclusive multiclass | C raw logits + cross-entropy | Label smoothing for justified regularization; cost weights for declared policy | Softmax before cross-entropy; missing rare-class slices |
| Multilabel | C independent logits + per-label BCE with logits | Asymmetric/focal variants with label-specific evidence | Softmax across labels that may co-occur |
| Regression | MSE for conditional mean; MAE for median | Huber for bounded outlier influence; quantile/pinball for asymmetric targets | Choosing by validation loss without matching action cost |
| Counts / positive targets | Poisson, Gamma, or Tweedie deviance when assumptions fit | Distributional NLL with learned uncertainty | Ignoring support, dispersion, zero inflation, or link semantics |
| Forecast intervals | Pinball loss per quantile or a distributional likelihood | CRPS or interval objectives for a full predictive distribution | Crossing quantiles and uncalibrated intervals |
| Ranking | Pointwise, pairwise logistic/hinge/BPR, or listwise softmax | Choose from slate structure, feedback bias, and serving policy | Assuming lower surrogate loss guarantees NDCG or utility |
| Retrieval / embeddings | InfoNCE, contrastive, or triplet objective | Hard-negative mining with leakage and false-negative controls | Batch composition silently defines the task |
| Segmentation | CE/BCE plus Dice, Tversky, or focal term when justified | Balance component gradient scales and validate small classes | Optimizing background pixels or summing incomparable scales |
| Detection | Classification/objectness plus box-regression or IoU-family loss | Focal classification for dense easy-negative imbalance | Unversioned assignment/matching rules |
| Language models | Masked token cross-entropy from raw logits | Preference/distillation objectives as separately evaluated stages | Padding in the loss, wrong causal mask, or train/eval tokenizer drift |
| Distillation | Temperature-scaled KL plus optional hard-target loss | Tune temperature and term weights under task metrics | Comparing unscaled gradients or inheriting teacher errors |
Change one assumption and make the tradeoff visible.
Use the worked chapter as your prediction, then change one input in the lab and explain why the output moves. The lab runs predefined TypeScript and never executes learner code, so the focus stays on the concept rather than setup.
Robust-loss influence lab
Increase one residual and compare how MSE, MAE, and Huber objectives react.
Assumption: Residuals [0.2, −0.5, 0.8, r]; MSE is mean squared error and Huber δ=1.
Open nonvisual data table
| Item | Computed state | Interpretation |
|---|---|---|
| MSE | 25.23250 | outlier gradient ∝ 10.0 |
| MAE | 2.87500 | bounded gradient magnitude |
| Huber δ=1 | 2.49125 | linear outlier tail |
Trace the complete operating path.
Carry the chapter’s central idea into the system: A loss chooses which mistakes drive learning; an evaluation contract decides whether that learning is useful. The path below shows where that reasoning needs an artifact, a test, an owner, or a fallback.
- 01
Turn “Write the tensor contract” into a written invariant and a failing test.
- 02
Promote robust_losses.py from illustration to a versioned implementation with representative fixtures.
- 03
Use “Evaluate the actual decision” as the release checkpoint: Hold the split and model budget fixed while comparing losses on calibration, thresholded utility, slices, stability, and compute—not training loss magnitude across unlike objectives.
- 04
Serve or materialize behind explicit stable gradients, representative batch composition, and compute that fits the training window.
- 05
Join telemetry to mature outcomes and retain a rollback path.
Observability
Join service health, input quality, prediction distributions, slice behavior, and mature outcomes by exact version.
Cost
Measure storage, preprocessing, compute, queueing, and human review under a representative arrival pattern.
Failure modes
The team uses MSE for every target, applies softmax before a numerically stable logits loss, weights rare examples twice, and assumes a lower training loss guarantees a better product decision. Add a detector, owner, mitigation, and stop condition for this class of failure.
Alternatives
Compare a rule, a simpler statistical baseline, and a different system boundary before adding model complexity.
Explain the contract, not just the vocabulary.
Loss-selection benchmark
Compare plausible objectives for one imbalanced classifier and one regression, ranking, or segmentation task under a fixed evaluation contract.
- Target/output shapes and logits/probability contract
- At least three objective ablations
- Gradient-scale, masking, and numerical-stability checks
- Independent decision metrics and calibration analysis
- Failure-slice recommendation with rollback criteria
Staged hints
Failure diagnosis
- If offline numbers look impossible, audit prediction-time availability before model code.
- If quality is sound but the contract fails, measure the exact serialized and served path.
- If one slice regresses, preserve that slice as a permanent release gate.
Record an honest attempt to unlock the reference solution.
Read primary material with a purpose.
PyTorch loss functions
35 min · last checked 26 Aug 2026
↗RECOMMENDED · Official docsscikit-learn regression metrics and losses
25 min · last checked 26 Aug 2026
↗RECOMMENDED · PaperFocal Loss for Dense Object Detection
30 min · last checked 26 Aug 2026
↗DEEP DIVE · INDEXFind a primary implementation or benchmark
30–90 min · verify task, dataset, and evaluation match
↗Return to the opening failure.
The training contract records target encoding, logits/probability expectations, reduction, weights, masking, numerical stability, gradient-scale diagnostics, and the separate evaluation metrics that gate release.
For this lesson, the release evidence is the worked code example, a hand-checked formal result, deterministic simulation output, a ≥80% checkpoint, the production rubric, and a named fallback. The course resolves when the system can produce a small network built from forward pass through backpropagation and optimization.