Loss Functions: Measuring "How Wrong," and Why We Minimize It
Lesson 5 used squared error as a stand-in for "how wrong" and moved on. That was a placeholder. The loss function is the single most consequential choice in a learning system, because it is the only thing that tells training what "better" means. Change the loss and you change the model you get — from the same data, the same shape, and the same optimizer.
An optimizer is obedient and literal. It does not know about houses, patients, or fraud; it knows one number and it makes that number small. So whatever you put in that number becomes the model's definition of success — including the parts you didn't mean. Write a loss that punishes big misses hardest, and you get a model that sacrifices typical accuracy to appease outliers. Write a loss that treats every mistake as equal, and you get a model indifferent to the one mistake that would cost you your job. The loss function is where your intent gets translated into arithmetic, and it is the only place that translation happens. Everything downstream is machinery faithfully serving whatever you wrote here.
We keep predicting house prices, but this lesson needs a dataset with an awkward row in it, because that's where loss functions start disagreeing with each other. Five houses, prices in thousands of euros:
| House | Size (m²) | Sale price (k€) | Note |
|---|---|---|---|
| A | 60 | 180 | ordinary flat |
| B | 85 | 200 | ordinary flat |
| C | 95 | 210 | ordinary flat |
| D | 100 | 220 | ordinary flat |
| E | 110 | 900 | penthouse — or a typo |
House E is the interesting one. Four houses live between 180 and 220; one sits at 900. Whether E is a genuine luxury sale or a data-entry mistake, your choice of loss decides how much attention the model pays to it. We'll watch that happen with real arithmetic.
What this lesson delivers
- Exact vocabulary: loss (one example), cost (whole dataset), objective (what we minimize) — and why people blur them.
- The four-part contract a function must satisfy to be usable as a loss.
- The standard regression losses — MSE, MAE, Huber — and a worked case where MSE and MAE pick different best models from identical data.
- Why accuracy cannot be a loss function, and what log loss (cross-entropy) does instead.
- How to read a loss as a statement of values: outliers, asymmetric costs, imbalanced classes.
1 · Loss, cost, objective — getting the words straight
Three words circulate for closely related ideas, and papers use them loosely. The distinction that actually matters is one example versus all examples.
Takes a single true value and a single prediction, returns one non-negative number. For house A alone: how bad was that one guess? Squared error, absolute error, and log loss are all per-example losses.
Average the per-example loss over all n examples. Notice what the notation says: the data is fixed, so J depends only on θ — the parameters. This is precisely the "error is a function of the parameters" move from Lesson 5, now written properly. J is the height on the error landscape.
In practice almost everyone says "loss function" for both, and "the loss" for the dataset-wide number you watch during training. Scikit-learn, textbooks, and library docs all mix them. You will not be misunderstood if you follow the crowd — but keep the two levels straight in your head, because per-example is where the design decisions live and dataset-wide is what the optimizer sees. The word objective function is the widest term: the thing being minimized, which is usually the cost plus extras such as a regularization penalty (Lesson 13).
2 · The contract: what makes a function usable as a loss
You cannot pick any formula you like. A loss has a job to do inside a training loop, and that job imposes four requirements. Three are about being meaningful; the fourth is about being optimizable, and it's the one beginners never anticipate.
Here is the shape of the problem. The optimizer only ever learns about the loss locally — it feels the slope under its feet and steps downhill. A loss that is flat over wide regions gives slope zero: no information, no direction, no step. A loss that jumps abruptly has no slope at all at the jump. So a measure can be a perfectly good report card for humans and still be a useless training signal. That gap — between what we want to evaluate and what we can optimize — is why the loss you train on and the metric you report are often two different functions. Hold that thought; it explains half of the practical decisions in Part V.
3 · Regression losses — and a genuine disagreement
Regression predicts a number, so the residual y − ŷ is available directly and the only question is how to price it. The two classics differ in one detail — square or absolute value — and that detail changes the model you get.
Both satisfy the contract: non-negative, zero when perfect, sign-blind. MSE prices a mistake by its square, so a 100k miss costs 100 times a 10k miss. MAE prices it linearly, so a 100k miss costs exactly 10 times a 10k miss. That's the whole difference, and it is not a small one.
Strip the problem to its bones. Use the simplest model that exists — one parameter, no features: predict the same constant c for every house. (ŷ = c. It's a legitimate model family; every value of c is a member.) The question: which c is best for our five prices — 180, 200, 210, 220, 900?
Two candidates suggest themselves: the mean (342) and the median (210). Score both under both losses, in k€:
| Candidate c | Residuals (y − c) | MSE | RMSE | MAE |
|---|---|---|---|---|
| 342 (mean) | −162, −142, −132, −122, +558 | 78,016 | ≈ 279 | 223.2 |
| 210 (median) | −30, −10, 0, +10, +690 | 95,440 | ≈ 309 | 148.0 |
Read the bold entries. MSE prefers 342. MAE prefers 210. Same data, same model family, same optimizer — and the two losses hand you different models. This is not a rounding artifact or a quirk of these five numbers; it is a theorem: the constant minimizing squared error is always the mean, and the constant minimizing absolute error is always the median.
Look at what 342 means as a prediction: it is above four of the five houses. If someone asks "what does a flat here cost?", answering "342k" is wrong for 80% of the market — MSE bought that by chasing the penthouse, because the squared penalty for missing a 900k house by 558 is so enormous that dragging the prediction up is worth ruining the other four. The median answer, 210k, describes four houses well and is badly wrong about one.
Neither is objectively correct. The right choice depends on what a mistake costs you. If you're a bank whose exposure grows with the square of the error, MSE's caution about the huge miss is exactly right. If you're publishing a typical-price estimate, MAE's answer is the honest one. What you cannot do is stay neutral — you pick a loss, and by picking it you have answered the question whether you meant to or not.
Squaring makes the loss sensitive to outliers; absolute value makes it robust to them. "Sensitive" and "robust" are the technical terms and neither is a compliment or an insult — a genuine 900k penthouse is signal worth chasing, and a mistyped 900k is noise worth ignoring. The loss cannot tell those apart. Only you can, which is why "look at your data before choosing your loss" is real advice rather than a platitude.
Huber loss — refusing to choose
In words: behave like MSE for small errors, like MAE for large ones. Inside a band of width δ around the truth it squares (smooth, well-behaved gradients near the answer); beyond that band it goes linear, so a wild outlier contributes a bounded pull instead of dominating. The two pieces are built to meet with matching slope, so the result is smooth enough to optimize.
δ is the threshold that declares "errors bigger than this are probably outliers, stop chasing them." Training does not learn it — you set it before training. By the Lesson 5 test (does the training procedure change this number by looking at the data? No), δ is a hyperparameter. With our house data, δ = 50 would treat the four ordinary flats quadratically and the penthouse's 690 miss linearly. Getting δ right needs the same tuning machinery as any hyperparameter (Lesson 21) — which is the honest cost of Huber's flexibility.
| Loss | Penalty grows | Outliers | Optimal constant | Smooth everywhere? | Reach for it when |
|---|---|---|---|---|---|
| MSE / L2 | quadratically | dominate | mean | yes | big errors are genuinely disproportionately costly; data is clean |
| MAE / L1 | linearly | bounded influence | median | no — kink at zero error | outliers or bad rows are present; you want typical-case accuracy |
| Huber | quadratic then linear | bounded influence | between the two | yes (by construction) | you want MAE's robustness without MAE's kink; you can afford to tune δ |
Three reasons stack up. Smoothness — the squared function has a clean slope everywhere, including at zero error, whereas |·| has a corner there that optimizers must special-case. Closed form — with a linear model, minimizing MSE has an exact algebraic solution (you can solve for the best w, b directly, no iteration), and the landscape is that single friendly bowl from Lesson 5. A probabilistic justification — if you assume the noise around the true line is Gaussian, then the parameters that make the observed data most likely are exactly the ones that minimize squared error. MSE isn't an arbitrary pick; it's the answer to a specific, and often reasonable, assumption about how the world is noisy. When that assumption is wrong — heavy tails, corrupted rows — MSE's optimality goes with it.
4 · Classification losses — and why accuracy is disqualified
Now switch tasks. Predict a category, not a number: will this house sell within 30 days? Yes or no. The obvious way to score a classifier is accuracy — the fraction it gets right. It is the right thing to report and it cannot be used to train. Understanding why is one of the most clarifying things in this module.
Suppose the model outputs a score and you predict "yes" when it exceeds a threshold. Now nudge w by a hair. Almost certainly, no prediction flips — every house stays on the side of the threshold it was already on — so the mistake count is exactly unchanged. The loss is flat: slope zero, no direction to step. Keep nudging and eventually one house crosses over, and the count jumps by a whole unit, instantly. So the accuracy landscape is a staircase: flat plateaus where the gradient is zero and tells you nothing, separated by cliffs where the slope is undefined.
That violates requirement 4 outright. An optimizer dropped on a staircase has no idea which way to walk. (This measure has a name — 0–1 loss, one unit per mistake — and minimizing it directly is computationally intractable for most models.) We need a loss that is sensitive to small parameter nudges, which means a loss that cares not just whether the model was right but how confidently.
The fix: have the model output a probability p between 0 and 1 rather than a hard yes/no, then penalize it by how much probability it assigned to the truth. Small nudges to w now move p slightly, which moves the loss slightly. Smooth landscape, usable gradient.
Labels are y = 1 or 0, and p is the probability the model assigned to class 1. The bracket is a switch: when y = 1 the second term vanishes and the loss is −log(p); when y = 0 the first vanishes and it's −log(1 − p). Either way, it's the negative log of the probability the model gave to the correct answer — which is the whole idea, and the notation is just a way to write it without an if-statement.
Take a house that did sell quickly (y = 1) and score several possible model outputs (natural log):
| Model says p(sells) = | Verdict | Loss = −log(p) |
|---|---|---|
| 0.99 | confident and right | 0.01 |
| 0.90 | right | 0.11 |
| 0.60 | right but unsure | 0.51 |
| 0.50 | no opinion | 0.69 |
| 0.10 | wrong | 2.30 |
| 0.01 | confidently wrong | 4.61 |
| 0.001 | certain and wrong | 6.91 |
Two properties to notice. The penalty is continuous — every small change in p changes the loss, which is exactly the training signal accuracy couldn't provide. And it is brutally asymmetric toward confident errors: being sure and wrong (p = 0.01) costs about 40× being right with p = 0.90. As p → 0 with y = 1 the loss heads to infinity. Log loss doesn't just want correct answers; it wants honestly calibrated ones, and it punishes bluffing without mercy.
Four houses, all of which sold quickly (y = 1 for all four). Two models, both predicting "yes" whenever p > 0.5:
| Model | Predicted probabilities | Accuracy | Mean log loss |
|---|---|---|---|
| A — timid | 0.60, 0.60, 0.60, 0.60 | 4/4 = 100% | 0.51 |
| B — informative | 0.99, 0.99, 0.99, 0.51 | 4/4 = 100% | 0.18 |
Accuracy cannot tell these apart — both are flawless. Log loss says B is roughly three times better, and it's right to: B tells you three of these are near-certainties and one is a coin flip, which is actionable information. A just shrugs "probably" four times. This is the second reason log loss earns its place: beyond being optimizable, it grades the quality of the model's uncertainty, which a hard yes/no score throws away.
Log loss is not the only smooth-enough classification loss. Hinge loss takes a different stance: it wants every example on the correct side of the boundary by a comfortable margin, and once an example is safely past that margin it stops caring entirely — that example's loss is exactly zero and it exerts no further pull. Only the borderline cases shape the boundary. That "only the difficult points matter" behavior is the engine of Support Vector Machines, so we'll leave the details to Lesson 18. Mentioning it now makes a structural point: the loss is a slot, and swapping what goes in it gives you a recognizably different algorithm from the same ingredients.
5 · The loss as a statement of values
Everything so far points one direction: the loss is where you tell the system what to care about. Three situations where the default is quietly the wrong answer.
Structurally, all of this fits into the training loop as a single swappable component:
the loss is a plug-in — one line decides what "better" means
# Per-example losses: interchangeable, same signature
def mse_loss(y, y_hat): return mean((y - y_hat) ** 2)
def mae_loss(y, y_hat): return mean(abs(y - y_hat))
def log_loss(y, p): return -mean(y * log(p) + (1 - y) * log(1 - p))
loss = mse_loss # <-- change THIS and you change the model
theta = init_parameters()
for step in range(num_steps):
y_hat = model(X, theta) # the MODEL predicts (Lesson 5)
J = loss(y, y_hat) # the COST: our height (this lesson)
grad = slope_of(J, theta) # which way is downhill (Lesson 7)
theta = step_downhill(theta, grad) # adjust the PARAMETERS (Lesson 5)Nothing else in the loop knows or cares which loss you chose — and that's the point worth sitting with. model, slope_of, and step_downhill are unchanged, yet swapping mse_loss for mae_loss gave us 342 instead of 210 earlier. The loss is a small, isolated, and enormously consequential line.
6 · Limitations — what to keep honest about
- A low loss is not a good model. The loss is measured on the data you trained on, and a flexible enough model can drive it toward zero by memorizing. Low training loss with high unseen error is overfitting (Lesson 13) — the loss cannot detect this, because it never looks at unseen data.
- Loss values are not comparable across losses or datasets. "My loss is 0.4" means nothing without knowing which loss, in what units, on what data. MSE scales with the square of your target's units — measure prices in euros instead of thousands and your MSE grows a millionfold with no change in model quality. Only compare like with like.
- The loss encodes your assumptions whether you thought about them or not. Choosing MSE asserts that big errors are disproportionately costly and your data is clean. Choosing an unweighted loss asserts every example matters equally. Defaults are assertions too — silent ones.
- Smoothness is a constraint, not a virtue. We use log loss partly because 0–1 loss is unoptimizable, which means we are minimizing a surrogate for what we actually want. Usually the surrogate's minimum is close enough to the real goal. Not always — this gap is where "the model has great log loss but bad recall" comes from.
- No loss can distinguish signal from corruption. Our 900k house might be a genuine penthouse or a typo. MSE chases it, MAE ignores it, and neither is exercising judgment — they're applying arithmetic. Data inspection is your job, not the loss function's.
How this connects to the next lesson
The E box from Lesson 5 is now exact. You know what a loss must satisfy to be usable, you have the standard choices for regression and classification, and you've seen with your own arithmetic that the choice changes the answer. The landscape from Lesson 5 finally has a precise definition: the loss is what defines the height at every point in parameter space.
Which leaves exactly one slot in the loop — slope_of and step_downhill. We have a well-defined surface and no method for descending it. Lesson 7 — Optimization intuition supplies that method: what a gradient is (intuitively — the direction of steepest increase), why stepping against it reduces the loss, what the learning rate controls and how it fails in both directions, and when the walk gets stuck. After Lesson 7 the template is complete, and Lesson 10 builds the first real algorithm on it.
Exercises & reflection
Pen and paper again. Exercise 2's arithmetic in particular is worth doing by hand — the numbers are the argument.
Pick a loss for each scenario and say in one sentence why:
- Predicting a house's price from a public dataset you know contains data-entry errors.
- Predicting whether a loan will default, where the bank needs a usable probability, not just a yes/no.
- Predicting daily bakery demand, where under-baking loses a customer permanently and over-baking wastes flour.
- Predicting delivery time, where being 60 minutes late is far worse than six times being 10 minutes late.
Show discussion
(1) MAE or Huber. Known-corrupt rows are exactly the case where MSE's squaring lets bad data steer the model; robustness is worth more than smoothness here. (2) Log loss. It requires and rewards calibrated probabilities — the bank needs to distinguish a 5% default risk from a 40% one, which a hard label discards. (3) An asymmetric loss weighted so under-prediction costs more; the two mistakes have genuinely different business costs, so a symmetric loss would misrepresent the problem. (4) MSE. This is the case its critics forget — when large errors really are disproportionately costly, quadratic growth is a feature, not a bug. Note that (1) and (4) are the same task with opposite answers: the loss follows the cost structure, never the task type.
Take our five prices (180, 200, 210, 220, 900) and remove house E. For the remaining four, compute the mean and the median. Then state, without further arithmetic, what constant MSE would choose and what constant MAE would choose — and compare the gap between them to the 342-vs-210 gap we found with E included. What does that tell you about which loss "notices" the penthouse?
Show solution
Four prices 180, 200, 210, 220: mean = 810/4 = 202.5; median = (200+210)/2 = 205. So MSE would choose 202.5 and MAE 205 — a gap of 2.5, versus a gap of 132 with the penthouse included. Nearly all of that 132 came from MSE alone: MAE's answer barely budged (205 → 210) when E was added, while MSE's leapt (202.5 → 342). That's what "sensitive" versus "robust" means quantitatively, and it's the sharpest way to see the difference: on clean data the two losses roughly agree, and they diverge precisely to the extent your data has extremes in it.
Three houses with true labels y = 1, 0, 1 (sold fast / didn't / sold fast). The model outputs p = 0.8, 0.3, 0.2. Compute each per-example log loss and the mean. Useful values: −log(0.8) ≈ 0.22, −log(0.7) ≈ 0.36, −log(0.2) ≈ 1.61. Then answer: at a 0.5 threshold, what is the accuracy — and does it capture the same thing the loss does?
Show solution
House 1 (y=1, p=0.8): loss = −log(0.8) ≈ 0.22. House 2 (y=0, p=0.3): the correct class is 0, whose probability is 1 − 0.3 = 0.7, so loss = −log(0.7) ≈ 0.36. House 3 (y=1, p=0.2): loss = −log(0.2) ≈ 1.61. Mean ≈ (0.22 + 0.36 + 1.61)/3 ≈ 0.73. Accuracy: predictions at a 0.5 threshold are 1, 0, 0 versus truth 1, 0, 1 → 2/3 ≈ 67%. The two disagree in character: accuracy records one mistake and stops there, while the loss shows that mistake was a fairly confident one (1.61 dwarfs the two correct examples combined) and that house 2 was correct but hesitant. Same predictions, strictly more information.
In three or four sentences, explain to a fellow engineer why "minimize the number of misclassified examples" cannot be used to train a model, and what log loss changes about the situation. Use the words flat, slope, and confidence. Then say what accuracy is still good for.
A loss function prices a single prediction's mistake; averaging it over the dataset gives the cost J(θ), which — because the data is fixed — is a function of the parameters alone and therefore is the height of the Lesson 5 landscape. To be usable a loss must satisfy four conditions: lower means better, perfection sits at the minimum, direction-blind unless deliberately not, and smooth in the parameters. For regression, MSE squares residuals (sensitive to outliers, optimal constant = the mean, smooth and closed-form), MAE takes absolute values (robust, optimal constant = the median), and Huber interpolates via a hyperparameter δ — and on our five houses MSE and MAE genuinely disagreed, 342 versus 210, proving the choice is substantive rather than cosmetic. For classification, counting mistakes gives a staircase whose gradient is zero on the plateaus and undefined at the cliffs, so it cannot be optimized; instead we predict probabilities and use log loss, which is smooth, punishes confident errors savagely, and grades calibration that accuracy ignores entirely. The loss is where your priorities get encoded — asymmetric costs, rare classes, what counts as a bad day — and the defaults encode priorities too, just silently. Lesson 7 takes this well-defined surface and finally teaches us how to walk down it.