2. Regression

1. Learning goals

After these notes, you should be able to:

• recognize a regression task and distinguish it from classification;
• explain features, targets, predictions, residuals, slopes, and intercepts;
• distinguish simple, multiple, polynomial, and regularized regression;
• describe ordinary least squares algebraically and geometrically;
• calculate and interpret MAE, MSE, RMSE, and $R^2$;
• select a metric based on the real cost of mistakes;
• fit and test a model without leaking information;
• use residuals to investigate model assumptions;
• explain each important line in the notebook's Python workflow.

2. What, why, and when

What is regression?

Regression is a supervised-learning task family in which one or more inputs are used to predict a numerical output:

$$ f:\mathbb{R}^{p}\rightarrow\mathbb{R}, \qquad \hat y=f(\mathbf{x}). $$

• $\mathbf{x}=(x_1,\ldots,x_p)$ is a feature vector.
• $y$ is the observed target.
• $\hat y$, pronounced "y-hat," is the prediction.

Regression is not one algorithm. Linear regression, regression trees, random forests, gradient boosting, support-vector regression, and neural networks can all solve regression problems.

A number is not automatically a regression target. Labels encoded as $0,1,2$ still represent classification if those numbers are merely category names.

Why use regression?

Regression supports two related goals:

1. Prediction: estimate an unknown numeric outcome.
2. Explanation: estimate how the expected target changes with features.

These are not identical. A simple model may be preferred for interpretation, while a nonlinear ensemble may predict better. Strong prediction or association does not by itself demonstrate causation.

When is linear regression useful?

Use it as a serious model or baseline when:

• the target is numeric;
• an approximately additive relationship is plausible;
• coefficient interpretation matters;
• a transparent and fast baseline is useful;
• extrapolation will be limited and defensible.

Revise the features, loss, or model when:

• the mean relationship bends strongly;
• interactions dominate;
• a few outliers control the fit;
• error variance changes greatly with prediction size;
• important time, group, or spatial dependence is ignored;
• the target needs a specialized distribution, such as counts or proportions.

Examples

For stock prices, a random row split is normally unsuitable because time order matters. Use time-based validation and domain-aware baselines.

3. Regression vocabulary

For $n$ rows and $p$ features:

An error is theoretically the difference from the unknown population relationship. A residual is the observed, fitted sample equivalent. Machine-learning explanations often use the words loosely, but statistics distinguishes them.

Array shapes

$$ X\in\mathbb{R}^{n\times p},\quad y\in\mathbb{R}^{n},\quad \boldsymbol{\beta}\in\mathbb{R}^{p}. $$

Even one feature is normally passed to scikit-learn as a 2-D matrix with shape $(n,1)$. This explains the notebook's X = np.random.rand(100, 1).

4. Regression model families

4.1 Simple linear regression

$$ \hat y=\beta_0+\beta_1x. $$

Exactly one feature predicts the target. Example: price from area alone.

4.2 Multiple linear regression

$$ \hat y=\beta_0+\beta_1x_1+\cdots+\beta_px_p. $$

With price, area, rooms, and age, $\beta_{\text{area}}$ is the modeled price change for one additional area unit while holding the other included variables fixed. This is conditional association, not automatically a causal effect.

4.3 Polynomial regression

$$ \hat y=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3. $$

The graph can curve, but the model remains linear in its coefficients. It is ordinary linear regression applied to transformed features $x,x^2,x^3$. High degrees can overfit and extrapolate wildly, so choose the degree with validation.

4.4 Regularized regression

Regularization controls coefficient size and model variance.

Ridge

$$ \min_{\boldsymbol{\beta}} \left[ \frac1n\sum_i(y_i-\hat y_i)^2 +\lambda\sum_j\beta_j^2 \right]. $$

Ridge shrinks coefficients toward zero and is often helpful with correlated predictors.

Lasso

$$ \min_{\boldsymbol{\beta}} \left[ \frac1n\sum_i(y_i-\hat y_i)^2 +\lambda\sum_j|\beta_j| \right]. $$

Lasso can set some coefficients exactly to zero.

Elastic Net

$$ \min_{\boldsymbol{\beta}} \left[ \frac{1}{2n}\lVert y-X\boldsymbol{\beta}\rVert_2^2+ \lambda\left(\rho\lVert\boldsymbol{\beta}\rVert_1+ \frac{1-\rho}{2}\lVert\boldsymbol{\beta}\rVert_2^2\right) \right]. $$

It blends Lasso and Ridge. Scale numeric features before regularization so feature units do not arbitrarily control penalty strength.

5. Linear regression intuition

The notebook creates:

$$ y=3x+5+\varepsilon, $$

where $\varepsilon$ is random noise. The learner sees only the noisy points and tries to recover a slope near $3$ and intercept near $5$.

Slope

$$ \beta_1=\frac{\Delta\hat y}{\Delta x}. $$

If $\beta_1=2.92$, the predicted target increases by about $2.92$ units when $x$ increases by one unit.

Intercept

$$ \hat y\big|_{x=0}=\beta_0. $$

The intercept is not always meaningful. If $x=0$ is impossible or far outside the observed range, interpreting it is extrapolation.

Residual

$$ e_i=y_i-\hat y_i. $$

• $e_i>0$: underprediction;
• $e_i<0$: overprediction;
• $e_i=0$: exact prediction.

Standard regression minimizes vertical differences because the setup treats $x$ as observed and uncertainty as acting mainly in $y$. If both axes contain material measurement error, an errors-in-variables or orthogonal method may be better.

6. How the best line is found

6.1 Least squares

$$ \operatorname{SSE}=\sum_{i=1}^{n}(y_i-\hat y_i)^2, \qquad \operatorname{MSE}=\frac1n\sum_{i=1}^{n}(y_i-\hat y_i)^2. $$

OLS chooses coefficients with the smallest SSE. Dividing by $n$, $2n$, or not dividing does not change the minimizer; it only rescales the objective and gradient.

Why square residuals?

• signs cannot cancel;
• large mistakes receive more weight;
• the objective is smooth;
• linear least squares produces a convex quadratic.

Squared loss is also the maximum-likelihood loss when independent errors are normally distributed with constant variance.

6.2 Closed-form simple-regression solution

$$ \hat\beta_1= \frac{\sum_i(x_i-\bar x)(y_i-\bar y)} {\sum_i(x_i-\bar x)^2}, \qquad \hat\beta_0=\bar y-\hat\beta_1\bar x. $$

The numerator measures co-movement; the denominator measures variation in $x$. Equivalently:

$$ \hat\beta_1=r_{xy}\frac{s_y}{s_x}. $$

The fitted line with an intercept passes through $(\bar x,\bar y)$.

6.3 Matrix form

With a column of ones for the intercept:

$$ \hat{\boldsymbol{\beta}}=(X^\top X)^{-1}X^\top y. $$

This textbook expression assumes invertibility. Software should not explicitly calculate that inverse. QR, singular-value decomposition, numpy.linalg.lstsq, and pseudoinverses are more stable and can handle rank deficiency.

For $X$ shaped $(n,p)$, SVD-based least squares commonly costs $O(np^2)$ when $n\ge p$. The notebook statement "matrix inversion is $O(n^3)$" mixes sample and feature dimensions.

6.4 Gradient descent

For

$$ J(\boldsymbol{\beta})=\frac1n\lVert y-X\boldsymbol{\beta}\rVert_2^2, $$

$$ \nabla J=-\frac{2}{n}X^\top(y-X\boldsymbol{\beta}), \qquad \boldsymbol{\beta}^{(t+1)} =\boldsymbol{\beta}^{(t)}-\alpha\nabla J. $$

Ordinary linear least-squares loss is convex. A tiny learning rate mainly converges slowly; it is not trapped by local minima. A very large rate can overshoot and diverge.

7. Train, validation, and test data

The notebook creates a reproducible 80/20 split: