Ridge Regression and Regularization

Why least squares overfits with many or correlated features, the ridge objective and its closed-form solution, how the penalty λ shrinks coefficients (SVD view and effective degrees of freedom), the bias–variance effect, standardisation and the unpenalised intercept, the Bayesian MAP interpretation, and comparison with the lasso (L1) and elastic net.

Why regularise?

Ordinary least squares is unbiased, but its variance explodes when:

there are many features relative to samples ( $p$ close to or larger than $n$ ),
features are strongly correlated (multicollinearity), so $X^\top X$ is nearly singular.

The coefficients become huge, unstable and overfit noise. Regularisation adds a penalty on coefficient size, accepting a little bias for a large drop in variance.

The ridge objective

\hat{\boldsymbol\beta}^{\text{ridge}} = \arg\min_{\boldsymbol\beta}\ \sum_{i=1}^n\left(y_i - \beta_0 - \sum_j\beta_jx_{ij}\right)^2 + \lambda\sum_{j=1}^p\beta_j^2, \qquad \lambda \ge 0.

The intercept is not penalised (centre $y$ or include an unpenalised column).
Equivalent constrained form: minimise SSE subject to $\sum\beta_j^2 \le t$ .

Closed form

With centred data, setting the gradient $-2X^\top(\mathbf y - X\boldsymbol\beta) + 2\lambda\boldsymbol\beta$ to zero:

\hat{\boldsymbol\beta}^{\text{ridge}} = (X^\top X + \lambda I)^{-1}X^\top\mathbf y.

$X^\top X$ has non-negative eigenvalues; adding $\lambda I$ shifts each up by $\lambda$ , so the matrix is always invertible for $\lambda > 0$ . Ridge was invented precisely to fix singular or ill-conditioned $X^\top X$ .

How shrinkage works: the SVD view

With $X = UDV^\top$ :

X\hat{\boldsymbol\beta}^{\text{ridge}} = \sum_{j}\mathbf u_j\,\frac{d_j^2}{d_j^2 + \lambda}\,\mathbf u_j^\top\mathbf y.

Compared with OLS (factor 1), each principal direction is shrunk by $\frac{d_j^2}{d_j^2 + \lambda}$ — strongly when $d_j$ is small, i.e. in directions where the data has little variance and estimates are noisiest. The effective degrees of freedom $\operatorname{df}(\lambda) = \sum_j\frac{d_j^2}{d_j^2 + \lambda}$ falls smoothly from $p$ to 0.

Ridge coefficient paths: smooth shrinkage, no coefficient becomes exactly zero.

Bias–variance effect

$\lambda$	Bias	Variance	Model
$0$	low (OLS is unbiased)	high	possibly overfit
moderate	some	much lower	usually best test error
$\to\infty$	high	→ 0	predicts the mean (underfit)

Training error increases monotonically with $\lambda$ ; test error is typically U-shaped. Choose $\lambda$ by cross-validation.

Bayesian interpretation

With Gaussian noise $\varepsilon \sim N(0, \sigma^2)$ and a prior $\beta_j \sim N(0, \tau^2)$ , the posterior mode (MAP estimate) is the ridge solution with $\lambda = \sigma^2/\tau^2$ . A stronger prior belief that coefficients are small (smaller $\tau$ ) means larger $\lambda$ .

Ridge vs lasso vs elastic net

	Ridge (L2)	Lasso (L1)	Elastic net
Penalty	$\lambda\sum\beta_j^2$	$\lambda\sum\lvert\beta_j\rvert$	$\lambda_1\sum\lvert\beta_j\rvert + \lambda_2\sum\beta_j^2$
Closed form	yes	no (coordinate descent)	no
Exact zeros (feature selection)	no	yes	yes
Correlated features	shares weight among them	tends to pick one arbitrarily	groups them
Bayesian prior	Gaussian	Laplace	mixture

The corners of the L1 region make zero coefficients likely; the round L2 region does not.