The Scientific Python Stack: Equipment for the Loop You Already Understand

What this lesson delivers

• Why plain Python lists are the wrong container for numeric ML, and what NumPy's array fixes.
• Vectorization: expressing a whole-dataset computation as one array operation — and why it is 10–100× faster, not just shorter.
• Broadcasting: the rule that lets arrays of different shapes combine, and how it removes loops you'd otherwise write.
• The shape discipline — the single habit that prevents most bugs you will meet from here to the end of the course.
• Pandas' DataFrame: why a labelled table of mixed columns is a different tool from an array, and where each belongs.
• Matplotlib: drawing the loss curve and the scatter plot that turn numbers into the diagnostics from Lesson 7.
• One end-to-end pass: CSV on disk → DataFrame → array → the training loop → a plotted loss curve.

1 · Why not just use plain Python?

Python already has lists, loops, and arithmetic. A house dataset is just a list of numbers; a gradient is just a sum. Why introduce a library at all? The honest answer has two halves, and both are worth seeing before you accept the tool.

Half one — lists are slow for numbers, structurally

A Python list is a list of objects. The number 200 inside a list is not eight bytes of value sitting next to its neighbours; it is a full Python object — a boxed integer with a type tag, a reference count, and a pointer — and the list holds a pointer to it, somewhere else in memory. Adding two lists element by element means, for every single element: follow a pointer, unbox the object, check its type, do the addition, box the result, store a new pointer. That per-element overhead dwarfs the actual arithmetic.

A NumPy array is the opposite: a single block of raw numbers, all the same type, laid out contiguously in memory — the same thing a C or Fortran array is. Arithmetic over it runs as a tight loop in compiled C, one machine instruction per element, with the CPU streaming cache-friendly memory and often doing several elements per instruction. The difference is not stylistic. It is the difference between an interpreter and a compiler, applied a million times.

The exact numbers depend on your machine; the order of magnitude does not. Two orders of magnitude is the gap between a training run you watch finish and one you abandon. And it compounds: the gradient loop from Lesson 7 does this kind of arithmetic on every step, thousands of times.

Half two — a dataset is not a bag of numbers

The speed argument gets you NumPy. But real data arrives as a table: rows are houses, columns are named quantities of different kinds — a floating-point size, an integer bedroom count, a text neighbourhood, a price with missing values scattered through it. A raw numeric array is the wrong shape for that: it has no column names, insists every cell share one type, and has no idea what a missing value is. That is Pandas' job. And once you have numbers, you need to see them — the loss curve, the scatter of points — which is Matplotlib's.

2 · NumPy: the array is the whole idea

Everything in NumPy is organised around one object, the ndarray (n-dimensional array). Learn its three defining attributes and most of the library follows.

creating arrays — the handful of constructors you will actually use

import numpy as np           # the universal alias; everyone writes np

prices = np.array([180, 200, 220])   # from a Python list  -> shape (3,)
prices.shape                        # (3,)
prices.dtype                        # dtype('int64')

X = np.array([[60, 2],                # a 2-D array: 3 houses, 2 features each
              [95, 3],                #   column 0 = size m2, column 1 = bedrooms
              [120, 4]])
X.shape                             # (3, 2)   <- rows, then columns

np.zeros(5)                        # [0. 0. 0. 0. 0.]   handy for initialising
np.ones((2, 3))                    # a 2x3 block of ones
np.arange(0, 1, 0.25)             # [0.   0.25 0.5  0.75]   like range, but floats
np.linspace(0, 1, 5)              # [0.  0.25 0.5  0.75 1. ]  5 evenly spaced points

Indexing and slicing — grabbing rows, columns, and subsets

selecting parts of an array without copying loops

X[0]            # first row:            [60  2]
X[0, 1]         # row 0, column 1:      2       (a single value)
X[:, 0]         # EVERY row, column 0:  [ 60  95 120]   (all the sizes)
X[:, 1]         # EVERY row, column 1:  [ 2  3  4]      (all the bedroom counts)
X[0:2]          # first two rows (a slice, like a list)

# Boolean masking: select by a condition, no loop, no if.
prices[prices > 190]              # [200 220]  -> keeps only elements passing the test
prices[prices > 190] = 999        # ...and you can assign into the selection too

The comma inside the brackets is the thing to notice: X[:, 0] means "all rows, column zero", and it hands you an entire column as a 1-D array in one expression. In plain Python that is a loop; here it is punctuation. That colon-comma idiom — pulling a whole column out to feed the model — is one you will type hundreds of times.

3 · Vectorization & broadcasting: the loop that vanishes

This is the section that matters. Vectorization is the single skill that separates NumPy code that is fast and readable from NumPy code that is neither. The idea is a mental discipline as much as a technique.

Arithmetic between arrays is element-wise by default. This is the foundation everything else builds on:

a = np.array([1, 2, 3])
b = np.array([10, 20, 30])

a + b            # [11 22 33]   pairs up, adds each pair
a * b            # [10 40 90]   element-wise product (NOT matrix multiply)
a ** 2           # [1 4 9]      squares each element
a - 1            # [0 1 2]      the 1 is applied to every element -> broadcasting

The payoff: the Lesson 7 gradient, with no loop

Here is where the theory and the tool meet. In Lesson 7 you computed, by hand, the gradient of MSE for the line model ŷ = w·x + b. The math was:

Watch what those two formulas become in NumPy. Every sum-over-i disappears into a single array expression:

the same gradient, two ways — contrast them line for line

# --- the loop a beginner writes (correct, but slow and noisy) ---
def gradient_loop(x, y, w, b):
    n = len(x)
    dw, db = 0.0, 0.0
    for i in range(n):
        y_hat = w * x[i] + b            # predict one house
        error = y[i] - y_hat            # its residual
        dw += x[i] * error              # accumulate the two sums by hand
        db += error
    return -2/n * dw, -2/n * db

# --- the vectorized version (fast, and reads like the math) ---
def gradient(x, y, w, b):
    y_hat = w * x + b                   # ALL predictions at once   -> array
    error = y - y_hat                   # ALL residuals at once     -> array
    dw = -2 * np.mean(x * error)      # one product, one mean     -> scalar
    db = -2 * np.mean(error)          # one mean                  -> scalar
    return dw, db

The two functions compute the identical numbers. But the second has no for, no index i, no running accumulators — and on a million rows it is roughly a hundred times faster. Read the vectorized version again and notice it is almost a transcription of the formula: y - y_hat is the residual y − ŷ, np.mean(x * error) is the average of xᵢ(yᵢ − ŷᵢ). This is the point of the whole lesson. When your NumPy reads like your mathematics, the code is easy to trust; when you're hand-managing indices and accumulators, it is easy to get subtly wrong.

M = np.array([[1, 2],
              [3, 4],
              [5, 6]])          # shape (3, 2)

M.sum()               # 21          everything, to one number
M.sum(axis=0)         # [ 9 12]     collapse the ROWS -> one total per column
M.sum(axis=1)         # [ 3  7 11]  collapse the COLUMNS -> one total per row

4 · Pandas: the labelled table

NumPy holds numbers beautifully and knows nothing about what they mean. Real data needs names, mixed types, and a story for missing values. That is the DataFrame — think of it as a spreadsheet or a database table that lives in Python, or as a dictionary of equal-length columns where each column may have its own type.

loading and inspecting a dataset — the first five minutes with any data

import pandas as pd            # the universal alias, like np

df = pd.read_csv("houses.csv")    # a whole CSV file -> a DataFrame, one line

df.head()             # first 5 rows      -> always look before you leap
df.shape              # (1000, 4)         -> rows, columns, same convention as NumPy
df.columns            # the column names
df.info()             # types + how many non-null values per column (spot missing data)
df.describe()         # count, mean, std, min/max per numeric column -> a fast sanity check

selecting columns, filtering rows, handling the missing

df["price"]                    # one column       -> a Series
df[["size_m2", "bedrooms"]]    # several columns   -> a smaller DataFrame

df[df["price"] > 190]         # rows passing a condition (boolean masking, as in NumPy)
df[df["bedrooms"] >= 3]["price"].mean()   # chain: filter, pick a column, summarise

df.isna().sum()               # how many missing values per column
df = df.dropna()              # drop rows with any missing value...
df["size_m2"].fillna(df["size_m2"].mean())   # ...or fill them with the column mean

X = df[["size_m2", "bedrooms"]].to_numpy()   # features  -> shape (n, 2)
y = df["price"].to_numpy()                    # target    -> shape (n,)

# From here on it is pure NumPy: X and y feed the exact loop from Lesson 7.

5 · Matplotlib: making the numbers visible

Lesson 7 told you the loss curve is your primary diagnostic instrument — that its shape tells you whether the learning rate is healthy, too small, or diverging. Matplotlib is how you actually draw it. Two plot types carry almost everything you'll need in this course.

the loss curve — the plot you learned to read before you could draw it

import matplotlib.pyplot as plt   # the universal alias: plt

# Suppose training filled this list, one loss value per step (see Section 6).
plt.plot(loss_history)                # x = step index (0,1,2,...), y = the losses
plt.xlabel("step")
plt.ylabel("cost J")
plt.title("Loss curve")
plt.show()                            # render the figure (or savefig to a file)

That handful of lines produces the exact curve from Lesson 7 — the steep initial drop, then the long flattening tail of a healthy run, or the jagged bouncing of an α set too high. You now have both halves: last lesson taught you what the curve means; this line teaches you how to see it. Reading the loss curve stops being a metaphor and becomes something on your screen.

the scatter plot — look at your data before you model it

plt.scatter(X[:, 0], y)              # feature (size) on x, target (price) on y
plt.xlabel("size (m2)")
plt.ylabel("price (k EUR)")

# overlay the line the model learned, to SEE the fit
xs = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
plt.plot(xs, w * xs + b, color="red")   # the fitted line y = wx + b
plt.show()

6 · Putting the stack together: one honest pass

Everything so far, assembled into the shape every project in this course takes. This is the Lesson 7 training loop — unchanged — now fed by real data through Pandas and NumPy, and reporting itself through Matplotlib. Read it as the skeleton you'll flesh out from Lesson 10 onward.

load → prepare → train → plot — the whole stack in one screen

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. LOAD + PREPARE  (Pandas): the workshop
df = pd.read_csv("houses.csv")
df = df.dropna()                              # deal with missing values honestly
x = df["size_m2"].to_numpy()               # hand clean numbers to NumPy
y = df["price"].to_numpy()                 # shapes: both (n,)

# 2. TRAIN  (NumPy): the engine -- this is the Lesson 7 loop, verbatim
w, b = 0.0, 0.0                            # start anywhere on the landscape
alpha = 0.0001                             # learning rate (needs scaling; see note)
loss_history = []                             # we will plot this afterwards

for step in range(1000):
    y_hat = w * x + b                         # MODEL   -> all predictions   (Lesson 5)
    error = y - y_hat                         # residuals
    J     = np.mean(error ** 2)             # LOSS: MSE, one number       (Lesson 6)
    loss_history.append(J)

    dw = -2 * np.mean(x * error)          # GRADIENT, vectorized        (Lesson 7)
    db = -2 * np.mean(error)
    w  = w - alpha * dw                       # STEP: the update rule        (Lesson 7)
    b  = b - alpha * db

# 3. REPORT  (Matplotlib): the instrument
plt.plot(loss_history)                        # is the curve healthy? (Lesson 7)
plt.xlabel("step"); plt.ylabel("cost J")
plt.show()

7 · Limitations — what to keep honest about

How this connects to the next lesson

Exercises & reflection

These are meant to be typed into a Python session and run. The stack is learned at the keyboard, not on the page — predict each answer before you run it, then check.

def mse_loop(y, y_hat):
    total = 0.0
    for i in range(len(y)):
        total += (y[i] - y_hat[i]) ** 2
    return total / len(y)

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)