Optimization Intuition: How a Model Walks Downhill
Lesson 5 gave you a landscape whose height is the error and whose coordinates are the parameters. Lesson 6 made that height exact. Two slots in the training loop are still empty: which way is down, and how far do I step. This lesson fills them. By the end you will have run gradient descent by hand, on paper, and watched it converge — and watched it explode.
Every algorithm in Part III and Part IV needs some way to find good parameters, and one method does the job for nearly all of them. Gradient descent is not one algorithm's implementation detail — it is the mechanism by which models learn, from the linear regression you'll build in Lesson 10 to the trillion-parameter systems this course deliberately stops before. Learn it once here, in its simplest possible setting, and every later "how does this model train?" question has the same answer.
There is a second reason, and it's the practical one. When training goes wrong — the loss becomes NaN, the loss plateaus at a terrible value, the loss bounces around and never settles — the cause is almost always in this lesson. Engineers who skipped the intuition here spend their careers changing random settings and hoping. Engineers who have it read the loss curve and know what to change.
You are standing somewhere on a hillside. Thick fog — you can see nothing beyond your own boots. You want to reach the lowest point in the valley. What can you actually use?
You cannot see the valley floor, so you cannot walk straight to it. You cannot see the whole terrain, so you cannot plan a route. What you can do is feel the ground directly beneath your feet and notice which direction tilts downward. So you take a step that way. Then, from your new position, you feel again, and step again.
That is the entire algorithm. It is almost insultingly simple, and the reason it matters is that the fog is not a metaphor for ignorance we could remove with better tools — it is a permanent, structural fact. In a model with a thousand parameters, "seeing the whole landscape" would mean evaluating the loss at an impossible number of points. The local slope is the only information that is ever cheap enough to obtain, so every practical training method is built out of it.
What this lesson delivers
- Why brute-force search and exact algebra both fail, and what that leaves us with.
- What a derivative is, operationally — a number you can measure by nudging, no calculus required.
- What a gradient is once there is more than one parameter, and why we step against it.
- The update rule, run by hand over six steps on real numbers — including why the steps shrink on their own.
- The learning rate: a table where the same problem converges, oscillates forever, or explodes depending on one number.
- The four ways descent gets stuck, and which of them you can actually do something about.
- Batch vs stochastic vs mini-batch, and how you decide when to stop.
1 · Why we can't just find the answer directly
Before accepting a method that gropes around in fog, an engineer should ask the obvious question: why not just compute the best parameters? There are two straightforward strategies, and it is worth seeing precisely how each one dies.
Strategy A — try every combination
The parameters are numbers. Pick a range, chop it into steps, try every combination, keep the one with the lowest cost. For a single parameter this is completely reasonable: 100 candidate values, 100 evaluations of J, done in a millisecond. The trouble is what happens when you add a second parameter, and a third.
| Parameters | Grid points (100 values each) | Time at 1 billion evaluations/second |
|---|---|---|
| 1 | 100 | instant |
| 2 | 10,000 | instant |
| 3 | 1,000,000 | ~0.001 s |
| 5 | 10,000,000,000 | ~10 s |
| 10 | 1020 | ~3,000 years |
| 50 | 10100 | longer than the universe has existed |
Each new parameter multiplies the work by 100. This is the curse of dimensionality, and you will meet it again in Lesson 15 and Lesson 26. Notice how modest the failure point is: a plain linear model on 50 features — a small, ordinary, unglamorous model — has 51 parameters and is already permanently out of reach. Brute force is not slow. It is impossible.
Worse: 100 values per parameter is a very crude net. The true optimum will sit between grid points, so even after paying the full combinatorial price you get an approximation. Gradient descent has no such ceiling — it can walk to any precision you like, because it isn't restricted to a pre-chosen set of candidates.
Strategy B — solve it with algebra
The cost J(θ) is just a function. At the bottom of a smooth valley the slope is zero, so set the derivative to zero and solve for θ. This is honest, classical mathematics, and for one specific case it works beautifully: linear regression with squared error has a closed-form solution — the normal equation — which we will meet properly in Lesson 10.
We are giving up an exact answer computed in one shot, and accepting an approximate answer computed by repeated small improvements. In exchange we get a method that is general (works for almost any model and loss), scalable (never needs the whole dataset at once), and simple (a handful of lines). That trade is the reason machine learning at scale is possible at all. It is also why "training" is a process you watch rather than a computation you wait for — the loss curve on your screen is this walk, plotted.
2 · The slope: a number you can measure by nudging
You do not need calculus to use a derivative. You need to know what it is, and there is a definition that requires nothing but arithmetic and honesty.
Nudge the parameter by a tiny amount h. See how much the cost changed. Divide by the size of the nudge. That ratio is the slope: how much the cost changes per unit of parameter change. Calculus gives you exact formulas for this ratio without doing the nudge — that's all calculus adds. The meaning is entirely contained in the nudge.
Two properties carry all the weight, and both are about the sign rather than the size:
Both cases are captured by one rule: move in the direction opposite to the sign of the slope. Subtracting the slope does exactly that, automatically — subtracting a negative number increases the parameter, subtracting a positive one decreases it. No if statement is required, and that is why the update rule in Section 4 is a single line.
Here is the dataset for this lesson — three houses, prices in thousands of euros: 180, 200, 220. Use the simplest model there is, the constant model from Lesson 6: predict the same number c for every house, with MSE as the loss.
Stand at c = 100 and evaluate. Residuals are 80, 100, 120; squares are 6,400, 10,000, 14,400; the mean is J(100) = 10,266.67.
Now nudge to c = 101. Residuals become 79, 99, 119; squares 6,241, 9,801, 14,161; the mean is J(101) = 10,067.67.
Negative, so increasing c reduces the cost — move right. The exact derivative here is −200; our one-unit nudge got within half a percent using two evaluations and one subtraction. You just computed a gradient without calculus. Everything that follows is this measurement, repeated.
For this constant model the exact slope is dJ/dc = 2(c − 200), where 200 happens to be the mean of the three prices. Check it at c = 100: 2(100 − 200) = −200. ✓ It also tells you something for free — the slope is zero exactly when c = 200, so the mean is the bottom of this valley, which is the Lesson 6 theorem showing up again from a different direction.
We will use this exact formula for the rest of the lesson because it keeps the arithmetic clean. Do not lose the point, though: the formula is a shortcut for the nudge, not a different concept. Real libraries compute exact derivatives symbolically because it's faster and more accurate — but if they didn't, nudging would still work.
3 · The gradient: slope in every direction at once
One parameter gives a curve and a single slope. Real models have many parameters, and then "which way is downhill" is a genuine question, because you can move in many directions.
The gradient is the list of slopes, one per parameter. Each entry (a partial derivative) answers a self-contained question: "if I nudge this parameter and hold all the others still, how does the cost change?" Collect those answers into a vector and you have the gradient — one number for each dial you can turn.
1. The gradient points in the direction of steepest increase. Not decrease — increase. This trips people up constantly. Out of every direction you could step, the gradient names the one that raises the cost fastest. Which is exactly what we want, because the fastest way down is then the exact opposite: −∇J. That minus sign is the whole of gradient descent.
2. The gradient is purely local. It describes the ground under your boots and says nothing whatever about the terrain ten metres away. It cannot see the valley floor, cannot tell you how far away it is, and cannot warn you about a cliff. Every limitation in Section 6 is a consequence of this single fact — the fog is built into the method, not into our implementation of it.
For the line model from Lesson 5 (ŷ = w·x + b) with MSE, the two partial derivatives work out to:
Take them on trust for now — Lesson 11 derives both from scratch and implements them in NumPy. But read what they say, because it's the sort of sanity check worth developing. Both are built from the residuals y − ŷ: if every prediction is perfect, both sums are zero, the gradient vanishes, and training stops. That is precisely the behaviour you'd demand. And notice that ∂J/∂w weights each residual by that house's x — a big house missed badly pulls the slope harder than a small house missed equally badly, which is geometrically exactly right.
4 · The update rule — and a descent you can run by hand
We now have a direction. One thing is missing: the gradient tells you which way to go, but says nothing about how far. Nothing in it is measured in "steps". So we supply that ourselves, as a number we choose.
Read it left to right: the new parameters are the old parameters, minus a small multiple of the gradient. The minus turns steepest ascent into steepest descent. The multiplier α (alpha) is the learning rate — a positive number you choose, typically between 0.0001 and 0.5, controlling how big a step you take in that direction.
Repeat until the walk stops making progress. That's it. That is the algorithm that trains almost everything.
Same three houses (180, 200, 220), same constant model, exact gradient 2(c − 200). Start deliberately badly at c = 100 and set α = 0.25. Each row is one application of the rule:
| Step | c | Cost J | Gradient 2(c−200) | Step taken (−α·grad) | New c |
|---|---|---|---|---|---|
| 0 | 100 | 10,266.7 | −200 | +50 | 150 |
| 1 | 150 | 2,766.7 | −100 | +25 | 175 |
| 2 | 175 | 891.7 | −50 | +12.5 | 187.5 |
| 3 | 187.5 | 422.9 | −25 | +6.25 | 193.75 |
| 4 | 193.75 | 305.8 | −12.5 | +3.125 | 196.875 |
| 5 | 196.875 | 276.4 | −6.25 | +1.5625 | 198.4375 |
Six lines of arithmetic took a terrible guess (100) to within 1.6 of the optimum (200), and the cost fell from 10,266 to 276. Do at least two of these rows yourself with a pen — the mechanism becomes obvious in a way that reading cannot deliver.
Three things in that table deserve to be pulled out, because each is a general property rather than an accident of these numbers.
gradient == 0 would loop forever.the training loop, now with every slot filled
# Lesson 5 gave us model + parameters. Lesson 6 gave us loss.
# This lesson supplies the last two lines.
theta = init_parameters() # start anywhere on the landscape
alpha = 0.25 # the learning rate: our step size
for step in range(num_steps):
y_hat = model(X, theta) # the MODEL predicts (Lesson 5)
J = loss(y, y_hat) # the COST: our height (Lesson 6)
grad = gradient(J, theta) # which way is UP (this lesson)
theta = theta - alpha * grad # step the other way (this lesson)
if norm(grad) < tolerance: # flat ground: we're done (Section 7)
breakThat is the complete skeleton of supervised learning. Everything in Parts III and IV changes what model and loss are; this loop stays put. It's worth appreciating how little there is here — the mystique of "training a model" resolves into four lines and a convergence check.
5 · The learning rate: one number that decides everything
α is not a detail. It is the difference between a model that trains, a model that never converges, and a model that overflows to NaN in nine steps. Our little problem is simple enough that we can see all three outcomes exactly.
Let g = c − 200 be the gap to the optimum. The update is c ← c − α·2g, so the new gap is g − 2αg = (1 − 2α)·g. Every step multiplies the gap by the same fixed number. Whether training succeeds is therefore decided entirely by whether that multiplier sits inside −1 and +1 — a question about α and nothing else.
| Learning rate α | Gap multiplier | Behaviour | Trace from c = 100 |
|---|---|---|---|
| 0.005 | 0.99 | Converges, painfully slowly | 101 → 102 → 103 → … (~450 steps) |
| 0.25 | 0.5 | Healthy — halves the gap each step | 150 → 175 → 187.5 → 193.75 |
| 0.5 | 0 | Lands exactly, in one step | 200 ✓ |
| 0.75 | −0.5 | Overshoots, but closes in | 250 → 175 → 212.5 → 193.75 |
| 1.0 | −1 | Oscillates forever, never converges | 300 → 100 → 300 → 100 … |
| 1.5 | −2 | Diverges — explodes | 400 → −200 → 1,000 → −1,400 |
Sit with the last two rows. The data is fine. The model is fine. The gradient is computed perfectly correctly at every step. And training still fails completely — producing garbage in the α = 1.5 case — because of a single constant. When someone says their loss became NaN, this is the first thing to check, and it is usually the answer.
The gradient is trustworthy only near where it was measured — it describes the tilt under your boots. Take a small step and that description still roughly holds, so you land lower. Take an enormous step and you leap clear across the valley and partway up the far slope, landing higher than you started. Now you are further out, so the slope there is steeper, so the next leap is bigger still. Each step amplifies the last and the whole thing runs away to infinity within a few iterations.
So the failure mode isn't that the gradient lied. It's that you took a local truth and applied it globally. The learning rate is precisely your statement about how far you're willing to trust a local measurement.
Landing exactly on the answer in one step looks like the ideal setting, and it is — for this problem. That value works only because it precisely matches the curvature of this particular bowl. Change the dataset and it's wrong; add a second parameter, whose valley has different curvature in a different direction, and no single α is perfect for both. Never read a "best learning rate" off one example. Practical advice: start around 0.01, watch the loss curve, and adjust by factors of 3 or 10 — not by small tweaks, since the useful range spans orders of magnitude. Lesson 21 makes this a systematic search rather than a craft.
The loss curve is your instrument for this, and it is genuinely diagnostic:
NaN / inf — α is far too large; the run has diverged. Stop it. Divide α by 100 and restart. (If that doesn't fix it, check for unscaled features — see below.)6 · When the walk goes wrong
Descent has no map. Four situations follow from that, and it's worth knowing which are real threats and which are folklore.
Local minima
A point that is lower than everything immediately around it, but not the lowest point overall. A small side-valley partway down the mountain. Descent walks into it, finds every direction leads up, concludes it has arrived — and stops, with a model that is worse than it needed to be. Because the gradient is local, this is undetectable from the inside: the flat ground of a local minimum and the flat ground of the true bottom feel identical.
Here is the reassuring part, and it's the reason this course can teach optimization in one lesson. A loss surface that is convex — bowl-shaped, with no side-valleys anywhere — has exactly one minimum, so any local minimum is the global one, and descent is guaranteed to find it. And it happens that linear regression with MSE and logistic regression with log loss are both convex.
Almost every model in this course sits on a convex surface. Set a sane learning rate and convergence to the best possible parameters is a mathematical guarantee, not a hope. The models with genuinely awkward, non-convex landscapes — full of local minima and saddle points — are neural networks, which is one reason deep learning needed a decade of specialised tricks to become trainable at all.
Keep this as a boundary marker for the course: classical ML mostly optimizes convex surfaces; deep learning does not. When Lesson 29 describes what changes past this course's edge, this is one of the changes.
Saddle points and plateaus
A saddle point is flat but goes up in one direction and down in another — a mountain pass. The gradient is zero, so descent slows to a crawl or stops, even though escape routes exist nearby. A plateau is a wide, nearly flat region: the gradient is tiny, the steps are tiny, and progress becomes glacial without ever formally stopping. Neither is common in the convex problems ahead, but both explain a loss curve that has gone flat while the loss is still bad.
Long narrow valleys — the one that will actually bite you
Suppose you predict house price from size in square metres (values around 60–200) and number of bedrooms (values around 1–5). The size parameter changes the predictions enormously for the same nudge; the bedroom parameter barely moves them. The cost landscape is therefore not a round bowl but a long, narrow, steep-walled ravine.
Descent behaves badly here for a very concrete reason: the steepest downhill direction points across the ravine, not along it. So the walk bounces from wall to wall, zigzagging, while creeping along the valley floor almost imperceptibly. Any α big enough to make progress along the valley is big enough to diverge across it. You are stuck between "too slow" and "explodes", with no good value in between.
The fix is not a cleverer optimizer. It is feature scaling: rescale every feature to a comparable range, and the ravine becomes a round bowl in which every direction has similar curvature and one learning rate serves all parameters. This is the single most common reason a correct implementation trains badly, and Lesson 12 covers exactly how to do it.
Plain gradient descent has an obvious weakness: it forgets. Each step is decided solely by the current slope, with no memory of the direction it has been travelling. Momentum fixes that by accumulating a running average of past gradients — like a heavy ball rolling downhill, it keeps going through small flat patches and its side-to-side zigzags cancel out while its along-the-valley motion accumulates. Adaptive methods (RMSProp, and the ubiquitous Adam) go further and maintain a separate effective learning rate per parameter, automatically taking smaller steps in steep directions and larger ones in flat directions. You will see optimizer="adam" everywhere in deep learning. They are refinements of the update rule you already know — not different ideas — and classical ML rarely needs them, so we note them and move on.
7 · How much data per step, and when to stop
One question remains unasked. Computing the gradient requires the residuals, which requires predictions for the data — but how much data? Using all of it every step is the obvious answer, and for a million rows it is also an expensive one, paid on every single step.
| Variant | Data per step | Gradient quality | Speed per step | Path to the bottom |
|---|---|---|---|---|
| Batch GD | the entire dataset | exact | slow | smooth, direct |
| Stochastic GD (SGD) | one example | very noisy | very fast | erratic, jittery |
| Mini-batch GD | a small group (32–256) | good approximation | fast | slightly noisy, efficient |
The trade is between the accuracy of each step and the number of steps you can afford. A noisy gradient from 32 examples still points roughly downhill, and you can take a hundred such steps in the time batch descent takes one — so you end up much further down. Mini-batch is the default everywhere, for that reason.
Counter-intuitively, a noisy gradient can be helpful. A jittery step can knock the walk out of a shallow local minimum or off a saddle point that exact batch descent would have settled into permanently. Randomness as an escape mechanism — it matters little for the convex problems in this course, but it is one of the reasons SGD works so well for neural networks.
Knowing when to stop
8 · Limitations — what to keep honest about
- Optimization finds the best parameters, not the best model. Our descent nailed c = 200 and the cost still sat at 266.7. A perfect optimizer on a badly chosen model family gives you the best of a bad set. "Training converged" and "the model is good" are unrelated claims.
- Converged does not mean correct. Descent will happily converge to a low training loss on a model that has memorised the data and will fail on anything new. The gradient never looks at unseen data. Nothing in this lesson can detect overfitting — that requires Lesson 19's methodology, not a better optimizer.
- There is no universally good learning rate. It depends on the loss, the model, the data's scale, and the batch size. Any number you're given — including 0.01 — is a starting point for a search, never a setting.
- Guarantees are conditional on convexity. "Gradient descent finds the global optimum" is true for the convex problems in this course and false in general. Repeating the guarantee outside its conditions is a common and confident error.
- Some models don't use gradient descent at all. Decision trees (Lesson 16) grow by greedy splitting; k-NN (Lesson 15) doesn't train at all; Naive Bayes (Lesson 17) counts frequencies directly. Gradient descent is dominant, not universal — it needs a continuous parameter vector and a differentiable loss, and not every model has either.
- Smooth in theory, awkward in floating point. Overflow, underflow, and vanishing precision are real; libraries contain a great deal of numerical care that clean mathematics doesn't reveal. This is one reason to use Scikit-learn in production and write from scratch only to learn.
How this connects to the next lesson
Module 1 is complete, and the training loop has no empty slots left. Lesson 5 defined the model and its parameters, and drew the landscape. Lesson 6 defined the loss that sets the landscape's height. Lesson 7 supplied the gradient that reads the local slope and the update rule that takes the step. Those four ideas — model, loss, gradient, step — are the entire conceptual content of supervised learning, and everything remaining in this course is a specific choice for one of the slots.
Which is why the next two lessons are about equipment rather than theory. Lesson 8 covers NumPy, Pandas, and Matplotlib: how to hold a dataset as an array, compute a whole gradient without writing a loop, and plot the loss curve you now know how to read. Lesson 9 tightens the mathematical vocabulary just enough — vectors, derivatives, probability — to make Lesson 10's derivations comfortable rather than intimidating. Then Lesson 10 and Lesson 11 take this exact loop, this exact update rule, and build a working Linear Regression in raw NumPy. You will recognise every line.
Exercises & reflection
Exercise 1 is the important one — running the loop by hand is what converts this from something you've read into something you know.
Same three houses (180, 200, 220), same constant model, gradient 2(c − 200). This time start at c = 300 — above the optimum — with α = 0.25. Compute four steps, writing down the gradient, the step taken, and the new c each time. Before you start, predict the sign of the gradient and which way c will move.
Show solution
Starting above the optimum, the gradient is positive (going right increases cost), so −α·grad is negative and c moves down. Step 0: c=300, grad = 2(100) = +200, step = −50 → 250. Step 1: grad = +100, step = −25 → 225. Step 2: grad = +50, step = −12.5 → 212.5. Step 3: grad = +25, step = −6.25 → 206.25.
Note the symmetry with the lesson's table: the same sequence of step sizes (50, 25, 12.5, 6.25), just in the opposite direction. The rule needed no modification for the "wrong side" case — the sign of the gradient handled it automatically. That is the whole reason we subtract the gradient rather than writing an if.
Same setup, starting at c = 100, but with α = 1.2. Compute three steps. What is the gap multiplier (1 − 2α), and what does the sequence of costs do? Then state the general condition on α that separates converging from diverging in this problem.
Show solution
Multiplier = 1 − 2.4 = −1.4. Gap starts at −100. Step 0: c = 100 − 1.2(−200) = 340 (gap +140). Step 1: c = 340 − 1.2(280) = 4 (gap −196). Step 2: c = 4 − 1.2(−392) = 474.4 (gap +274.4). The gap flips sign and grows 1.4× every step, so the cost rises without bound — in floating point this reaches inf, then NaN, within a few dozen iterations.
The condition is |1 − 2α| < 1, i.e. 0 < α < 1 for this particular problem. The general lesson is not the number 1 — it's that the threshold depends on the curvature of the loss surface, which changes with the model, the loss, and the scale of your features. That is exactly why the safe α must be found empirically rather than looked up.
You train four models. Diagnose each and say what you'd change:
- Loss drops fast for 50 steps, then flattens at a low value and stays there for 950 more.
- Loss decreases in a nearly straight, very gentle line and is still clearly falling at step 1,000.
- Loss jumps up and down between 400 and 900 for the whole run, with no trend.
- Loss goes 500, 2,000, 40,000,
inf,NaN.
Show discussion
(1) Healthy convergence. Nothing to fix in the optimizer — you could stop at step 200 and save the compute. If the final loss is still unacceptably high, the limit is the model family, not the training. (2) α too small. It's working, just slowly; multiply α by 10. Also consider running longer if compute allows. (3) α slightly too large — overshooting and bouncing across the valley without diverging. Divide by 3 or 10. If it persists at small α, suspect unscaled features creating a narrow ravine (Lesson 12). (4) Divergence. α is far too large. Divide by 100 and restart; if it still explodes, check feature scaling and look for extreme values or a bug in the gradient sign.
In four or five sentences, explain to a fellow engineer why gradient descent uses only local information, and what two consequences follow from that. Use the words local, step size, and guarantee. Then answer: why does a convex loss surface make one of those consequences disappear?
We cannot find good parameters by searching (100 values per parameter becomes 1020 combinations by ten parameters) or by algebra (closed-form solutions rarely exist and don't generalize), so we descend instead. The slope is a number you can measure by nudging a parameter and dividing the change in cost by the size of the nudge; its sign tells you which way is downhill, and the gradient collects one such slope per parameter into a vector that points toward steepest increase — hence the minus in the update rule θ ← θ − α∇J. Running that rule by hand on three house prices took a guess of 100 to within 1.6 of the optimum in six steps, with the steps shrinking on their own (50, 25, 12.5…) because the slope flattens near the bottom, and with the cost bottoming out at 266.7 rather than zero because the model shape, not the optimizer, sets that floor. The learning rate is the one genuinely dangerous knob: on the same problem, α = 0.25 converges cleanly, α = 1.0 oscillates forever, and α = 1.5 explodes — the gradient being a purely local truth that a large step applies globally. Descent can also stall in local minima, saddle points, plateaus, and long narrow ravines caused by unscaled features (fixable, Lesson 12) — but linear and logistic regression are convex, so for almost everything in this course a sane α guarantees the global optimum. Mini-batch gradients trade exactness for many more steps and win. Module 1 is now complete: model, loss, gradient, step — and Lesson 10 will build a real algorithm from exactly these four parts.