Roadmap › Part II · The Machine Learning Mindset › Module 2 · Tools & Just-Enough Math

Lesson 09 · Setup

The Math You Actually Need: Three Tools, No Textbook

Last lesson gave you the equipment; this one gives you the notation. To build Linear Regression from scratch you need exactly three pieces of mathematics — vectors (so a prediction over many features is one operation instead of a sum you write out by hand), derivatives (which turn the "nudge in the downhill direction" you already did in Lesson 7 into a formula), and probability (the language classification will later speak). This is not a maths course. It is a vocabulary lesson, and every symbol here is one you will type as NumPy within two lessons.

Reading: ~35 min Prerequisites: Lessons 5–8 · high-school algebra Code: each idea shown as both a formula and a np call

Why a math lesson, and why such a small one

Your instinct, as an engineer, may be to skip this and reach for the algorithm. Resist it for twenty minutes. From Lesson 10 onward every derivation writes a model as w·x + b, differentiates a loss to get a gradient, and — for classification — squeezes a score into a probability. If those three moves are fluent, the algorithms read like prose. If they are not, every lesson becomes a fight with notation instead of a lesson about ideas. This page exists so that fight never happens.

Equally important is what this lesson refuses to do. Your university will teach you linear algebra and calculus properly, with proofs. We are not competing with that — we are extracting the thin slice those subjects contribute to this course and presenting it as intuition plus NumPy. You will leave able to use the dot product and the derivative, not to prove theorems about them. That is exactly enough, and no more, on purpose.

What this lesson delivers

What a vector is in ML terms — one example, or one set of parameters, held as a list of numbers.
The dot product: why w·x + b is the single most-used expression in the course, and why it is one multiply-and-add.
Length and distance between vectors — the machinery k-NN and k-Means will run on.
The matrix as a whole dataset, and the matrix–vector product as "all predictions at once".
The derivative as slope — formalising Lesson 7's nudge — and the gradient as a vector of them.
Enough probability — conditional probability and Bayes' rule — to make Logistic Regression and Naive Bayes readable when they arrive.
For every one of these: the matching NumPy, so the math and the code are one thing in your head.

1 · The three questions, and the three tools that answer them

Rather than march through "linear algebra, then calculus, then probability" as disconnected subjects, anchor each to the concrete question it answers in this course. There are only three, and you have already met all three informally.

"How do I combine many features into one prediction?" — A house has size, bedrooms, age, distance-to-centre. The model weights each and sums them. That combine-and-sum is the dot product, and it is why we need vectors. Answered in Sections 2–4.

"Which way is downhill on the loss surface?" — Lesson 7 nudged the parameters to reduce error and watched the loss fall. The exact direction and size of that nudge is a derivative, and the collection of them across all parameters is the gradient. Answered in Section 5.

"How confident is this prediction?" — A spam filter shouldn't just say "spam"; it should say "94% spam". Turning scores into confidences, and reasoning about evidence, is probability. Answered in Section 6.

Keep the map, not the details

If at any point the symbols feel heavy, come back to this frame. Vectors answer combine, derivatives answer which way downhill, probability answers how confident. Every equation in the course is a variation on one of those three questions — and knowing which question an equation is answering is most of the battle.

2 · Vectors: one example is a list of numbers

You already used this in Lesson 8 without naming it. When you wrote a house as [60, 2] — sixty square metres, two bedrooms — you wrote a vector. There is nothing more mysterious to it than that.

Vector

A vector is an ordered list of numbers. In ML it plays two distinct roles, and keeping them straight is half the value of the word:

A feature vector — one example. The house x = (60, 2) is a point described by its features. A dataset is a pile of these, one per row. This is the vector you feed in.

A parameter vector — one model. The weights w = (w₁, w₂) — how much price responds to each of size and bedrooms — is also a vector. This is the vector you learn.

The number of entries is the vector's dimension. A house with four features is a 4-dimensional vector, a point in 4-D space. You cannot picture 4-D, and you never need to: the algebra that works in the 2-D you can picture works identically in 400-D. That is the quiet superpower of vector notation — it is written once and holds for any number of features, so a formula proven on a whiteboard with two features is already the formula your code runs on two hundred.

a vector is just a 1-D NumPy array — you have been making them since Lesson 8

import numpy as np

x = np.array([60, 2])          # a feature vector: one house, 2 features
w = np.array([3.0, 15.0])      # a parameter vector: price per m2, price per bedroom

x.shape                            # (2,)   <- dimension 2, a 2-D vector
x + x                              # [120   4]   add element-wise (Lesson 8)
2 * x                              # [120   4]   scale every entry

Two operations define a vector: you can add two of the same dimension (line them up, add entry by entry) and you can scale one by a single number (multiply every entry). Geometrically, adding vectors places one arrow's tail at the other's head; scaling stretches or flips an arrow. You will lean on the intuition, not the geometry — but it is worth knowing the arrows are there, because "the model is a point in parameter space and learning walks it downhill" (Lesson 7) is exactly that picture.

3 · The dot product: the one expression the whole course is built on

Here is the single most important piece of notation in classical ML. When a linear model predicts a price from a house's features, it multiplies each feature by its weight and sums the results:

prediction = w₁·x₁ + w₂·x₂ + … + wₙ·xₙ + b

Writing that sum out is tedious for two features and impossible for two hundred. The dot product is the notation that compresses the entire multiply-each-pair-and-add into two symbols.

Dot product

The dot product of two vectors of the same dimension, written w·x, multiplies matching entries and adds the results into a single number:

w·x = Σ_i wᵢ·xᵢ = w₁x₁ + w₂x₂ + … + wₙxₙ

Two vectors go in; one scalar comes out. That scalar is a weighted sum — a score. Almost every linear method in this course computes a score exactly this way.

With it, the whole linear model collapses to a single, dimension-independent expression — the one you will read a hundred times from Lesson 10 onward:

ŷ = w·x + b

Read it aloud: "the prediction is the dot product of the weights with the features, plus a bias." That one line is Linear Regression's model (Lesson 10), the score inside Logistic Regression (Lesson 14), and the signed distance inside an SVM (Lesson 18). Learn to see w·x + b and think "weighted vote of the features, plus an offset" and three future lessons are already half-understood.

the dot product is one call — and it replaces a loop, exactly like Lesson 8's gradient

x = np.array([60, 2])
w = np.array([3.0, 15.0])
b = 20.0

np.dot(w, x)          # 3*60 + 15*2 = 210.0   <- one weighted sum
w @ x                 # 210.0   the @ operator IS the dot product, and reads cleanly

y_hat = w @ x + b     # 230.0   the model, verbatim: w.x + b

# the loop it replaces -- correct, but the thing we are escaping
total = 0.0
for i in range(len(w)):
    total += w[i] * x[i]          # total ends at 210.0, slowly

Why w·x is a "weighted vote"

Think of each weight wᵢ as how strongly feature i gets to vote, and its sign as the direction of the vote. A large positive w on "size" says bigger pushes price up, hard; a negative w on "age" says older pulls it down. The dot product tallies every feature's vote into one number. When Lesson 10 "learns the weights", it is learning how much each feature's vote should count — and that is the entire content of training a linear model. The bias b is the baseline before any feature speaks.

Length and distance — the same idea, pointed at yourself

A dot product of a vector with itself gives a second tool you'll need the moment we reach k-NN and k-Means: a notion of size, and from it, distance between two points.

Norm and Euclidean distance

The length (or norm) of a vector is the square root of its dot product with itself — Pythagoras, in any dimension:

‖x‖ = √(x·x) = √(x₁² + x₂² + … + xₙ²)

The distance between two points is the length of the vector between them — subtract, then take the norm:

dist(a, b) = ‖a − b‖ = √(Σ_i (aᵢ − bᵢ)²)

That second formula is the entire mathematical content of k-Nearest Neighbors (Lesson 15): "which stored points are closest to this new one?" is answered by computing this distance to each. It reappears in k-Means (Lesson 24), which assigns each point to its nearest cluster centre. The algorithms differ in what they do with distance; the distance itself is this one line.

length and distance, straight from the norm

a = np.array([60, 2])
b = np.array([95, 3])

np.sqrt(a @ a)                 # 60.03...   the length of a
np.linalg.norm(a)              # 60.03...   same thing, the direct call

np.linalg.norm(a - b)          # 35.01...   Euclidean distance between the two houses
np.sqrt(np.sum((a - b)**2))     # 35.01...   the formula spelled out, to see it is the same

4 · Matrices: the whole dataset in one object

One house is a vector. A thousand houses stacked as rows is a matrix — and you have been holding datasets in exactly this shape since Lesson 8, where a (1000, 4) array meant a thousand rows of four features. The word "matrix" simply names that 2-D array and unlocks one operation that makes it powerful.

Matrix, in ML terms

A matrix is a rectangle of numbers — rows × columns. In this course it is almost always the design matrix X of shape (n, d): n examples down the rows, d features across the columns. Row i is the feature vector of example i. The whole dataset, one object.

Now the payoff. You know how to predict for one house: ŷ = w·x + b. To predict for all thousand, you do not loop — you take the matrix–vector product X w, which computes the dot product of every row with w in one operation and returns a vector of a thousand predictions:

ŷ = X w + b shapes: (n, d) · (d,) → (n,), then + b broadcasts

Matrix–vector product = "the dot product, applied to every row"

There is nothing new to learn here. X w is just "take row 1, dot it with w; take row 2, dot it with w; …" — the Section 3 dot product run once per example, bundled into a single call. This is vectorization (Lesson 8) at the level of the whole model: the loop over a thousand houses vanishes into compiled C, and the code that predicts for one house and the code that predicts for a million look identical. That is why the from-scratch algorithms stay short.

all predictions at once — the model, scaled to the whole dataset

X = np.array([[60, 2],        # 3 houses, 2 features -> shape (3, 2)
              [95, 3],
              [120, 4]])
w = np.array([3.0, 15.0])
b = 20.0

y_hat = X @ w + b     # [230.  350.  440.]  <- three predictions, one line, no loop
y_hat.shape           # (3,)   one prediction per row

The one rule that prevents every matrix bug: shapes must line up

A matrix product is only defined when the inner dimensions match: (n, d) times (d,) works because the d’s meet; the result takes the outer dimensions, (n,). Try (n, d) times (n,) and NumPy raises a shape error — which, as Lesson 8 insisted, is a gift. When a from-scratch algorithm throws, nine times in ten it is this: a w or an X transposed the wrong way. The fix is always the same — print both .shapes and read whether the inner dimensions touch. You will not need the full theory of matrix multiplication in this course; you will need this one alignment rule constantly.

5 · Derivatives: putting a number on "which way is downhill"

In Lesson 7 you minimised a loss by hand: you were somewhere on a bowl-shaped surface, you figured out which direction went down, and you stepped that way. The derivative is the piece of mathematics that computes that direction and its steepness exactly, instead of by eye. It is the formal name for the nudge you already performed.

Derivative

The derivative of a function at a point is its slope there — the rate at which the output changes as you nudge the input. Written df/dx or f′(x), it answers: "if I increase x by a tiny amount, how much, and in which direction, does f move?" Positive slope: f rises as x rises. Negative slope: f falls. Zero slope: flat — a peak, a valley, or a ledge.

That last case is the whole reason we care. A loss function is a bowl; its lowest point is where the model fits best; and at the bottom of the bowl the slope is zero. So "train the model" becomes "find where the derivative is zero", and "step downhill" becomes "step against the sign of the derivative". Three facts carry all of it:

Sign points the way. If the slope is positive, the function increases to the right, so to go down you move left — against the slope. This is precisely why the update rule subtracts: w ← w − α·(slope).

Size measures steepness. A big slope means a steep wall — a big nudge changes the loss a lot; a small slope means near-flat ground. This is why steps naturally shrink as you approach the bottom: the slope itself is fading.

Zero means stop. Slope zero is a flat spot — for the bowl-shaped losses in this course, the minimum. Gradient descent halts itself there because the update −α·0 is no move at all.

You need the derivatives of only a couple of shapes, and you can read them off intuitively rather than memorising rules. The one that matters most is the parabola, because squared error — the loss from Lesson 6 — is one:

The three derivatives this course actually uses

f(x) = x² → f′(x) = 2x — steep far from 0, flat at 0 (the bottom of the bowl)

f(x) = c·x → f′(x) = c — a straight line has one constant slope, its coefficient

f(x) = c (a constant) → f′(x) = 0 — flat everywhere; the bias term differentiates away

Notice x² has slope 2x: far to the right of zero the parabola climbs steeply (big positive slope), far to the left it drops steeply (big negative slope), and exactly at the bottom the slope is zero. That single picture — steep on the sides, flat at the minimum — is the shape of every loss surface you will descend.

From one derivative to the gradient

A real model has many parameters — a weight per feature, plus a bias. The loss depends on all of them at once, so "the slope" is not a single number but one slope per parameter: how the loss changes as you nudge w₁ alone, w₂ alone, b alone. Each of these is a partial derivative (written ∂J/∂w₁), and stacking them into a vector gives the gradient.

Gradient

The gradient ∇J of a loss J is the vector of its partial derivatives — one slope for each parameter, bundled together:

∇J = ( ∂J/∂w₁, ∂J/∂w₂, …, ∂J/∂b )

It has a beautiful geometric meaning: the gradient points in the direction of steepest increase of the loss. So minus the gradient points straight downhill — the direction of fastest decrease. That is the whole of "gradient descent": repeatedly step a little way along −∇J.

You have already computed a gradient

This is not new material dressed up — it is Lesson 7 and Lesson 8, named. The two formulas you used there,

∂J/∂w = −(2/n)·Σ xᵢ(yᵢ − ŷᵢ) ∂J/∂b = −(2/n)·Σ (yᵢ − ŷᵢ)

are the two components of the gradient of Mean Squared Error — and the 2 in each is precisely the 2x derivative of the squared term above. In Lesson 8 you wrote them as dw = -2 * np.mean(x * error) and db = -2 * np.mean(error). So you have already implemented the calculus of this section; Lesson 10 will simply derive those two lines from scratch, and now you have the vocabulary to follow every step.

The chain rule, in one sentence — the only rule you'll reach for repeatedly

Losses are functions of a prediction which is itself a function of the parameters — a chain. The chain rule says: to differentiate a nested function, multiply the slopes of each link. "How does the loss change with w?" = "how does the loss change with the prediction" × "how does the prediction change with w". That single idea is where the xᵢ factor in ∂J/∂w comes from — the prediction w·x changes with w at rate x. You do not need to master it here; recognise the phrase, because every from-scratch derivation from Lesson 10 on is one application of it.

6 · Probability: the language of confidence and evidence

Regression predicts a number and is done. Classification — is this email spam? is this tumour malignant? — should not answer with a bare yes/no but with a confidence: "94% spam." Probability is the mathematics of that confidence, and of reasoning from evidence. Two ML methods are built directly on it (Logistic Regression, Naive Bayes), and evaluation itself (Lesson 20) speaks it fluently, so a small, sturdy foundation pays off repeatedly.

Probability, the three facts that anchor everything

A probability P(A) is a number in [0, 1]: 0 = impossible, 1 = certain, 0.5 = a coin's-edge maybe.
The probabilities of all mutually exclusive outcomes sum to 1 — something must happen. (This is why a classifier's class probabilities are made to add to one.)
If two events are independent — one tells you nothing about the other — the probability of both is the product: P(A and B) = P(A)·P(B). (This product is the "naive" shortcut at the heart of Naive Bayes.)

Conditional probability — updating a belief with evidence

The idea classification lives on is the conditional probability P(A | B): "the probability of A given that we know B." A spam filter wants P(spam | the words in this email) — how likely spam is, given the evidence in front of it. The vertical bar reads "given"; it is how new evidence sharpens a guess.

Bayes' rule — flipping the condition around

Often you can easily measure P(evidence | cause) but actually want P(cause | evidence). Bayes' rule is the bridge between them:

P(A | B) = P(B | A) · P(A) / P(B)

In spam terms: P(spam | words) = P(words | spam) · P(spam) / P(words). You can estimate "how often these words appear in known spam" (P(words | spam)) and "how common spam is overall" (P(spam), the prior) directly from a labelled corpus — and Bayes turns those into the answer you wanted. This equation is the entire skeleton of Naive Bayes (Lesson 17); that lesson's only added ingredient is the independence shortcut above, used to make P(words | spam) computable.

From a score to a probability — where regression meets classification

Section 3 gave you a score, w·x + b, that ranges over all real numbers. A probability must live in [0, 1]. Something must squash the unbounded score into that interval — and that something is the sigmoid (logistic) function, the hinge on which Logistic Regression turns.

The sigmoid: an S-curve that turns any score into a probability

σ(z) = 1 / (1 + e^−z) where z = w·x + b

It maps a very negative score to near 0, a very positive score to near 1, and a score of 0 to exactly 0.5 — the point of maximum uncertainty. Feed the linear score through it and the number stops being an arbitrary tally and becomes an honest confidence. That is the whole move that converts Linear Regression's machinery into a classifier: same w·x + b, wrapped in a squashing function. Lesson 14 is essentially this one equation, taken seriously.

the sigmoid, and why it needs no loop — it broadcasts over a whole vector of scores

def sigmoid(z):
    return 1 / (1 + np.exp(-z))      # np.exp broadcasts: works on a number OR an array

sigmoid(np.array([-4.0, 0.0, 4.0]))
# [0.018  0.5   0.982]  <- a strong 'no', a shrug, a strong 'yes'

Why probability, and not just a threshold

You could classify by thresholding the raw score at zero and never mention probability. Two things break if you do. First, you lose the confidence — and a medical or fraud model that says "malignant, 51%" should be treated very differently from one that says "malignant, 99%". Second, you lose a principled loss to train against: once outputs are probabilities, there is a natural measure of wrongness (cross-entropy, Lesson 14) that punishes confident mistakes far more than hesitant ones, which is exactly the incentive you want. Probability is not decoration on classification; it is what makes classification trainable and trustworthy.

7 · Limitations — what to keep honest about

Guardrails to carry forward

This is intuition, not rigour. Real linear algebra and calculus courses prove what we have only motivated — and you should take those courses. What you have here is the working subset that makes this course readable, not a substitute for the mathematics itself.
Notation compresses; it does not simplify. w·x + b hides a sum over every feature, and X w hides a loop over every example. The brevity is the point — but never forget the loops are still there, running in C. When something is slow or memory-hungry, it is those hidden loops.
Linearity is an assumption, not a law. The dot-product model assumes the target is a weighted sum of features. When the true relationship curves or features interact, that assumption is wrong — which is what polynomial features (Lesson 12) and non-linear models (trees, Lesson 16) exist to fix.
Zero gradient is not always the global minimum. For the bowl-shaped losses in this course it is, which is why gradient descent works so cleanly here. In general (and in the deep learning beyond our scope) a flat spot can be a local dip or a saddle — a caveat worth filing away, not worrying about yet.
Probabilities from a model are estimates, not truths. A model that outputs "0.94" is reporting its confidence given its training data, which may be biased or too small. A calibrated-looking number can still be wrong — treat model probabilities as informed guesses, and Lesson 20 as where you learn to check them.

How this connects to the next lesson

Bridge to Lesson 10 — Linear Regression from scratch

Module 2 is now complete, and with it the whole runway. You have the equipment (Lesson 8: arrays, vectorization, broadcasting) and the vocabulary (this lesson: the dot product, the derivative, probability). Every tool the from-scratch lessons need is in your hands.

Lesson 10 opens Part III and builds a real Linear Regression from nothing. Watch how completely it is assembled from these two lessons: the model is ŷ = X w + b (Section 4's matrix–vector product), the loss is Mean Squared Error (Lesson 6, a parabola whose derivative you now know is 2x-shaped), the gradient is the vector of partials from Section 5, and the training loop is Lesson 7's descent written in Lesson 8's NumPy. Nothing new will be introduced except how the pieces click together — because you spent Module 2 making sure every piece was already familiar. That is the reward for a tools-and-notation module done patiently: the first real algorithm arrives feeling less like new material and more like a conclusion you were always heading toward.

Exercises & reflection

Do the by-hand parts on paper first, then check with a Python session — the notation sticks when your hand and the code agree.

Exercise 1 · Compute a prediction by dot product

A model has weights w = (3, 15) (price per m², price per bedroom) and bias b = 20. Compute the predicted price for a house x = (80, 3) as w·x + b, by hand. Then write the one line of NumPy that reproduces it.

Show solution

w·x = 3·80 + 15·3 = 240 + 45 = 285; add b = 20 → 305. In NumPy: np.array([3, 15]) @ np.array([80, 3]) + 20, which returns 305. Read it back as a weighted vote: size contributes 240, bedrooms 45, baseline 20.

Exercise 2 · Predict for a whole dataset, then check the shape

With the same w and b, and the design matrix X = np.array([[60, 2], [95, 3], [120, 4]]), write the single expression that predicts all three prices at once. Before running it, state the shape of the result and why.

Show solution

X @ w + b gives [230. 350. 440.], shape (3,). The product (3, 2)·(2,) has matching inner dimension 2, so it is legal, and the result takes the outer dimension — one prediction per row, (3,). The scalar b then broadcasts across all three. If you had written w @ X instead you'd get a shape error, because (2,)·(3, 2) misaligns — a reminder to read the inner dimensions.

Exercise 3 · Read a derivative as a direction

You are minimising a loss and, at your current weight, the slope ∂J/∂w comes out to +8. (a) Is the loss increasing or decreasing as w grows here? (b) To reduce the loss, should you increase or decrease w? (c) With learning rate α = 0.1, what is the new w if the old one was 5?

Show solution

(a) A positive slope means the loss increases as w grows. (b) So to go downhill you must decrease w — move against the slope. (c) The update is w ← w − α·(∂J/∂w) = 5 − 0.1·8 = 5 − 0.8 = 4.2. The subtraction is exactly what makes "against the slope" automatic — a positive gradient produces a step to the left.

Exercise 4 · Squash a score into a probability

A logistic model produces a raw score z = w·x + b = 2.0 for some email. (a) Roughly what probability of spam does the sigmoid assign — below, at, or well above 0.5? (b) What score z would correspond to exactly 0.5? (c) In one sentence, why can't we use the raw 2.0 as a probability directly?

Show solution

(a) σ(2.0) ≈ 0.88 — well above 0.5, a fairly confident "spam". (b) z = 0 gives exactly 0.5, the point of maximum uncertainty. (c) Because a probability must lie in [0, 1], and the raw score is unbounded — 2.0 is a fine score but a meaningless probability until the sigmoid squashes it into range.

Exercise 5 · Bayes on paper

A disease affects 1% of people (P(sick) = 0.01). A test is positive in 90% of sick people (P(+ | sick) = 0.9) but also in 5% of healthy people (P(+ | healthy) = 0.05). You test positive. Before computing, guess whether P(sick | +) is closer to 90% or to 15% — then use Bayes' rule to find out. (This is a famous intuition-breaker; the point is to feel why the answer surprises people.)

Show solution

Denominator P(+) = P(+|sick)P(sick) + P(+|healthy)P(healthy) = 0.9·0.01 + 0.05·0.99 = 0.009 + 0.0495 = 0.0585. Then P(sick | +) = (0.9·0.01) / 0.0585 = 0.009 / 0.0585 ≈ 0.154 — only about 15%, not 90%. The reason: the disease is so rare that the false positives from the huge healthy population outnumber the true positives from the tiny sick one. The low prior P(sick) dominates — which is exactly why priors matter, and why Naive Bayes keeps one.

One-paragraph summary to lock it in

Classical ML needs a thin, specific slice of mathematics, and it answers three questions. How do you combine features into a prediction? — with vectors and the dot product: a house is a feature vector, a model is a weight vector, and ŷ = w·x + b is a weighted vote of the features that reappears in Linear Regression, Logistic Regression, and SVMs alike; the same dot-product machinery, aimed at a vector minus another, gives the Euclidean distance that k-NN and k-Means run on, and stacked over rows becomes the matrix–vector product X w that predicts for a whole dataset in one call. Which way is downhill on the loss? — with the derivative, the slope whose sign points away from the minimum and whose size measures steepness, bundled across all parameters into the gradient ∇J, whose negative is the direction gradient descent steps — and you have already computed exactly this, as the -2 * np.mean(...) lines of Lesson 8. How confident is a prediction? — with probability: conditional probability P(A|B) and Bayes' rule underwrite Naive Bayes, while the sigmoid squashes an unbounded score w·x + b into a [0,1] confidence and turns regression's machinery into a classifier. None of this was rigorous, and none of it needed to be — it is the working vocabulary that makes Lesson 10 read like a conclusion rather than a surprise. Module 2 is done; the equipment and the notation are both in hand; Linear Regression from scratch is next.