Foundations — Regression vs Classification, Loss Functions and Evaluation

The supervised and unsupervised learning settings, regression vs classification, the training/validation/test workflow, empirical risk minimisation and common loss functions, and how models are evaluated — MSE, MAE, R², the confusion matrix, accuracy, precision, recall, F1, specificity, ROC curves and AUC — including the pitfalls of imbalanced data.

The learning problem

Supervised learning: given labelled examples $\{(\mathbf x_i, y_i)\}_{i=1}^n$ , learn a function $f$ with $f(\mathbf x) \approx y$ for new inputs.

Task	Target $y$	Examples	Typical models
Regression	continuous number	price, temperature, demand	linear/ridge regression, k-NN regression, trees, neural networks
Classification	discrete class	spam/ham, digit 0–9, disease yes/no	logistic regression, k-NN, naive Bayes, LDA, SVM, trees, neural networks

Unsupervised learning: only inputs $\{\mathbf x_i\}$ ; discover structure — clustering (k-means, hierarchical), dimensionality reduction (PCA), density estimation, anomaly detection.

Other settings: semi-supervised (few labels), self-supervised, reinforcement learning (rewards from actions).

Models, parameters and hyperparameters

Parameters are learned from data (regression weights, network weights).
Hyperparameters are set before training (k in k-NN, $\lambda$ in ridge, tree depth, learning rate) and chosen by validation — see cross-validation.
Parametric models have a fixed number of parameters (linear regression); non-parametric ones grow with the data (k-NN, decision trees).
Discriminative models learn $P(y \mid \mathbf x)$ or a decision boundary directly (logistic regression, SVM); generative models learn $P(\mathbf x \mid y)$ and $P(y)$ and use Bayes' theorem (naive Bayes, LDA).

Training, validation and test

Never tune on the test set — its error would no longer be an unbiased estimate.

Empirical risk minimisation: choose parameters minimising the average training loss $\frac1n\sum_i L(y_i, f(\mathbf x_i))$ , often plus a regularisation penalty. The real goal is low generalisation error on unseen data; a model that fits the training data too closely overfits.

Loss functions

Loss	Formula	Used by
Squared (L2)	$(y - \hat y)^2$	linear and ridge regression
Absolute (L1)	$\lvert y - \hat y\rvert$	robust regression (median-like)
0–1	$\mathbb 1[y \ne \hat y]$	the error rate itself (not optimised directly)
Log loss (cross-entropy)	$-[y\log p + (1-y)\log(1-p)]$	logistic regression, neural networks
Hinge	$\max(0, 1 - y\,f(\mathbf x))$ , $y \in \{\pm1\}$	SVM

Regression metrics

\text{MSE} = \frac1n\sum(y_i - \hat y_i)^2, \quad \text{RMSE} = \sqrt{\text{MSE}}, \quad \text{MAE} = \frac1n\sum\lvert y_i - \hat y_i\rvert,

R^2 = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum(y_i - \hat y_i)^2}{\sum(y_i - \bar y)^2}.

$R^2 = 1$ is a perfect fit; $R^2 = 0$ means no better than predicting $\bar y$ ; on test data it can be negative. Adding features never lowers training $R^2$ ; adjusted $R^2 = 1 - (1 - R^2)\frac{n-1}{n-p-1}$ penalises extra features.

Classification metrics

Confusion matrix (binary):

	predicted +	predicted −
actual +	TP	FN
actual −	FP	TN

Metric	Formula	Question it answers
Accuracy	$\frac{TP + TN}{N}$	how often correct overall?
Precision	$\frac{TP}{TP + FP}$	when it says "positive", how often right?
Recall / sensitivity / TPR	$\frac{TP}{TP + FN}$	how many positives does it find?
Specificity / TNR	$\frac{TN}{TN + FP}$	how many negatives does it clear?
False-positive rate	$\frac{FP}{FP + TN} = 1 - \text{specificity}$
F1 score	$\frac{2PR}{P + R} = \frac{2TP}{2TP + FP + FN}$	harmonic mean of precision and recall

ROC curve: plot TPR against FPR as the threshold varies; AUC = probability a random positive is scored above a random negative (0.5 = chance, 1 = perfect). It is threshold-independent and insensitive to class balance; on heavily imbalanced data the precision–recall curve is more informative.

Multi-class: compute per-class precision/recall and average them — macro (unweighted mean over classes) or micro (pool all TP/FP/FN; equals accuracy for single-label problems).