Model, Parameters, and Error
Lesson 4 gave you the words in passing — model, parameters, error. Now we make them exact. These three ideas are the atoms of every algorithm in this course. Once you can point at each one precisely and say what changes it, "training a model" stops being a phrase and becomes a mechanism you understand.
In Lesson 4 you saw the universal training loop: predict, measure error, adjust, repeat. Every word in that loop points at one of three things. The model is the thing that predicts. The parameters are the numbers that "adjust" changes. The error is what "measure" produces. Get these three crisp and the loop is no longer mysterious — it's just moving parameters to shrink error. By the end you'll be able to look at any model and answer three questions: what is fixed, what is learned, and what number are we trying to make small?
We keep the running example from Lesson 4 so the abstract terms always have something concrete to point at — predicting a house's sale price. The five rows again:
| Size (m²) | Bedrooms | Age (yrs) | Neighborhood | Sale price (€) |
|---|---|---|---|---|
| 60 | 2 | 30 | East | 180,000 |
| 85 | 3 | 12 | Center | 295,000 |
| 120 | 4 | 5 | Center | 430,000 |
| 45 | 1 | 40 | West | 132,000 |
| 100 | 3 | 8 | East | 350,000 |
To keep the math visible by hand, most of this lesson uses one feature — size — and the simplest model there is: a straight line.
What this lesson delivers
- A precise definition of a model as a family of functions — and why "choosing a model" means choosing a shape, not an answer.
- The clean split between parameters (learned) and hyperparameters (chosen by you).
- What error means for one prediction, and how single errors combine into one number for the whole dataset.
- The picture of parameter space — the landscape training walks across — which sets up loss (Lesson 6) and optimization (Lesson 7).
1 · What a model really is
Casually, "the model" means the finished predictor. But to understand training you need the sharper meaning: a model is a family of functions — a shape with empty slots. Training picks one specific member of that family by filling the slots with numbers.
The semicolon matters: x is the input you're given; θ (theta) are the knobs training turns. Fix the form, and every choice of θ gives a different concrete function. The model is the whole family; a trained model is one member of it.
Take just size as input and predict price. The simplest model is a straight line:
Here the form — "a line" — is fixed. The two slots w (slope, €per m²) and b (intercept, the base price) are the parameters. Pick w = 3,000 and b = 10,000 and you get one line; pick w = 3,500 and b = 0 and you get a different one. Every possible line is one member of this one model family. Training's job is to choose the member that fits the houses best.
When you pick a line, you are betting that price rises steadily with size — the same €per m² whether a flat is small or huge. That bet might be wrong: maybe the first few m² add little and large luxury homes jump in price. Choosing a richer shape — a curve (polynomial, Lesson 12), a tree (Lesson 16) — is choosing a different bet about how the world behaves. The model is your assumption about the shape of the answer; the data then fills in the details. This is why there is no single "best model": the right shape depends on the true pattern in the data.
The same idea scales straight up to all four features from the full table — just more slots:
Same shape (a weighted sum, a line in higher dimensions), just five parameters instead of two. We'll return to the one-feature line because you can draw it and do its arithmetic by hand.
2 · Parameters vs hyperparameters
Not every number in an ML system is learned. There are two kinds, and confusing them is one of the most common early mistakes. The distinction is simple: parameters are found by training; hyperparameters are set by you before training starts.
Ask: "Does the training procedure change this number by looking at the data?" If yes, it's a parameter. If you must fix it before training can even begin, it's a hyperparameter. In our line: w and b are set by fitting the houses → parameters. "Use a straight line rather than a degree-3 curve" is a decision you made first → a hyperparameter.
How many parameters does each model have?
- One-feature line w·size + b → 2 (w, b).
- Four-feature linear model → 5 (four weights + one intercept).
- Degree-2 curve on one feature a·size² + b·size + c → 3 (a, b, c) — and "degree = 2" is the hyperparameter that decided there'd be three.
Notice the pattern: the hyperparameter (the shape/degree) determines how many parameters exist. You choose the shape; the shape sets the number of slots; training fills them.
3 · Error — from one house to one number
A model with some parameters makes predictions. Those predictions are wrong by some amount. "Error" is how we make "wrong by some amount" into a number we can push down. Build it in two steps: first the error on a single prediction, then how to combine all of them into one score for the whole dataset.
Step 1 — error on a single prediction (the residual)
For one house it's just the gap between reality and the model's guess. Positive means the model under-predicted; negative means it over-predicted. This single number is the raw material of all error measurement.
Take the one-feature line ŷ = 3,000·size + 20,000 and run it on three houses from the table:
| Size | True price y | Prediction ŷ = 3000·size + 20000 | Residual y − ŷ |
|---|---|---|---|
| 60 | 180,000 | 200,000 | −20,000 |
| 85 | 295,000 | 275,000 | +20,000 |
| 120 | 430,000 | 380,000 | +50,000 |
Three gaps: the model over-prices the small house, under-prices the two larger ones. Now — how do we turn these three numbers into one score for "how good is this line?"
Step 2 — combine residuals into one dataset score
We can't just add the residuals: the −20,000 and +20,000 would cancel, making a badly-wrong line look perfect. We need every error to count as a cost regardless of sign. The standard fix is to square each residual (squares are always positive) and take the average. That's the Mean Squared Error.
n = number of examples; the Σ ("sum") runs over all of them. In words: square every residual, then average. One number that summarizes how wrong the whole model is. Lower is better; zero means every prediction is exact.
Using the three residuals above (−20,000, +20,000, +50,000):
The number itself is huge and in "euros-squared," which is hard to feel. That's why we often report its square root, the Root Mean Squared Error: √(1.1×10⁹) ≈ €33,000 — an average-sized miss, back in real euros. The raw MSE isn't meant to be intuitive; it's meant to be minimized. What matters is that a better line gives a smaller MSE.
Absolute value (|residual|) also kills signs and gives a perfectly valid error measure (Mean Absolute Error). Squaring is preferred for two reasons. First, it punishes big misses much harder than small ones — a €50,000 miss counts far more than five €10,000 misses — which usually matches what we want. Second, and decisively for the next two lessons, the squared function is smooth: it has a clean, everywhere-defined slope, which is exactly what the optimizer in Lesson 7 needs to know which way to nudge the parameters. This choice of "how to measure wrong" has a name — the loss function — and it's the entire subject of Lesson 6.
Here is the idea that unlocks everything ahead. The data is fixed. So once you write MSE out, the only things that can change it are the parameters. That means:
Different lines (different w, b) → different total error. Training is nothing more than searching for the w, b that make E(w, b) smallest. The model gave us predictions; the error turned a whole model into a single score; now that score is a landscape we can descend.
4 · Parameter space — the landscape training walks
This is the picture to carry into the next two lessons. Because error is a function of the parameters, we can imagine a map where each location is a choice of parameters and the height at that location is the error there.
Picture the two knobs w and b as the two directions on a flat floor (east–west, north–south). At every point on that floor, stack up the MSE that that pair of values produces as height. You now have a surface — a landscape of hills and valleys. A good model sits low (small error); a bad model sits high. Our line above (RMSE ≈ €33,000) is somewhere up a slope. Somewhere on that floor is the lowest point — the w, b with the smallest possible error. Training is the act of walking downhill to find it.
For a linear model with MSE, this landscape is a single smooth bowl — one valley, no false bottoms to get stuck in. That's a gift: wherever you start, downhill always leads to the answer. Not every model gives such a kind landscape (deep trees and, later, neural nets have rugged terrain), but linear regression's bowl is why it's the first algorithm we build — you can literally roll to the bottom. The rolling procedure has a name, gradient descent, and it's Lesson 7.
So the three words connect into one sentence you should be able to say from memory:
Where Lessons 6 and 7 plug in
You now have the shape of the whole training machine, with two slots left to fill — and they are precisely the next two lessons:
# The training loop from Lesson 4, now annotated with today's words
w, b = init_parameters() # start somewhere on the landscape
for step in range(num_steps):
y_hat = w * size + b # the MODEL predicts (this lesson)
E = mean((y - y_hat) ** 2) # the ERROR / loss = our height (Lesson 6)
grad = slope_of_E(w, b) # which way is downhill (Lesson 7)
w, b = step_downhill(w, b, grad) # adjust the PARAMETERS (this lesson)
# stop when E stops shrinking -> we're at the bottom of the bowlThree of the four lines are things you now understand exactly. Lesson 6 makes E rigorous (what "loss" can be, and why); Lesson 7 makes slope_of_E and step_downhill real (gradient descent). After that, every algorithm in the course is a variation on this template.
5 · Limitations — what to keep honest about
- The model can only be as good as its shape allows. If price truly curves with size, no w, b for a straight line will ever fit well — the lowest point of that bowl is still a bad line. Underfitting is a wrong-shape problem, not a training problem.
- Low training error is not the goal — low unseen error is. A shape flexible enough can drive training error to zero by contorting through every point, yet fail on new houses. This is overfitting (Lesson 13); the error landscape here is measured on training data only.
- Squared error is a choice, not a law. It punishes big misses hard and is swayed by outliers (one mistyped €0 house can dominate the MSE). Sometimes absolute error or another loss fits the goal better — the reason Lesson 6 exists.
- More parameters ≠ better. Adding slots lets a model fit more, but also memorize more. The count of parameters is a dial with two edges, revisited in the bias–variance trade-off (Lesson 13).
How this connects to the next lesson
You've now got the three atoms: a model (a shape with slots), its parameters (the numbers training fills), and error (one score that turns a whole model into a height on a landscape). We used squared error as a stand-in for "how wrong," but glossed over why that formula and what other choices exist. Lesson 6 — Loss functions makes the E box exact: what a loss function must do, the standard ones for regression and classification, and how the choice of loss quietly shapes what your model ends up caring about. Then Lesson 7 gives us the way to actually descend the landscape it defines.
Exercises & reflection
Pen-and-paper still — the arithmetic here is the whole point, because it makes the machinery physical.
Label each as a parameter (learned) or hyperparameter (chosen):
- The slope w in ŷ = w·size + b.
- The decision to use a degree-3 polynomial instead of a line.
- The maximum depth a decision tree is allowed to reach.
- The intercept b.
Show discussion
(1) Parameter — training sets it from the data. (2) Hyperparameter — you fix the shape before training. (3) Hyperparameter — a limit you impose on the training process. (4) Parameter. The test each time: does training change it by looking at the data? Yes → parameter.
Use the line ŷ = 3,500·size − 30,000 on the same three houses (sizes 60, 85, 120; true prices 180,000 / 295,000 / 430,000). Compute each prediction, each residual, and the MSE. Is this line better or worse than the one in the lesson (RMSE ≈ €33,000)? Report your answer as an RMSE for a fair comparison.
Show solution
Predictions: 3,500·60 − 30,000 = 180,000; 3,500·85 − 30,000 = 267,500; 3,500·120 − 30,000 = 390,000. Residuals (y − ŷ): 0; +27,500; +40,000. MSE = (0² + 27,500² + 40,000²)/3 = (0 + 756.25M + 1,600M)/3 ≈ 785M. RMSE = √785,000,000 ≈ €28,000. That's lower than €33,000, so this line is better — you've just compared two members of the same model family by their height on the error landscape, which is exactly what training does automatically.
Two models make these residuals on four houses. Model A: (+10k, −10k, +10k, −10k). Model B: (0, 0, 0, +25k). Which has lower total absolute error? Which has lower MSE? Explain in one sentence why squaring can crown a different winner than absolute error does.
Show solution
Absolute: A = 40k, B = 25k → B wins. Squared sums: A = 4·(10k)² = 400M; B = (25k)² = 625M → A wins. Squaring magnifies B's single large miss more than it penalizes A's many small ones, so the choice of error measure literally changes which model you'd pick — the reason loss selection (Lesson 6) is a real decision, not a formality.
In two or three sentences, explain what a single point on the (w, b) error landscape represents, what its height means, and what "training" is doing on that landscape. Use the words parameters, error, and downhill.
A model is a family of functions — a fixed shape (like a line w·size + b) with empty slots; choosing the model is choosing a bet about the shape of the answer. The slots come in two kinds: parameters (w, b), which training learns from data, and hyperparameters (like the degree of the shape), which you fix beforehand and which decide how many parameters exist. To judge a set of parameters we measure error: the residual y − ŷ for one house, combined across all houses into a single score such as Mean Squared Error. Because the data is fixed, that error is a function of the parameters alone — a landscape whose height is the error and whose lowest point is the best model. Training is walking that landscape downhill, and the next two lessons fill in what "downhill" means: the loss (Lesson 6) that defines the height, and the optimization (Lesson 7) that takes the steps.