Simple and Multiple Linear Regression

The linear model

$$y = \beta_0 + \beta_1x_1 + \dots + \beta_px_p + \varepsilon, \qquad \hat y = \hat\beta_0 + \hat\beta_1x_1 + \dots + \hat\beta_px_p.$$

"Linear" refers to linearity in the parameters: $y = \beta_0 + \beta_1x + \beta_2x^2$ (polynomial regression) and $y = \beta_0 + \beta_1\log x$ are still linear regressions.

Simple linear regression

Minimise $\text{SSE}(\beta_0, \beta_1) = \sum_i(y_i - \beta_0 - \beta_1x_i)^2$. Setting both partial derivatives to zero:

$$\hat\beta_1 = \frac{S_{xy}}{S_{xx}} = \frac{\sum(x_i - \bar x)(y_i - \bar y)}{\sum(x_i - \bar x)^2} = r\,\frac{s_y}{s_x}, \qquad \hat\beta_0 = \bar y - \hat\beta_1\bar x.$$

Properties of the OLS fit (with an intercept):

• residuals sum to zero, $\sum e_i = 0$, and are orthogonal to $x$: $\sum x_ie_i = 0$;
• the line passes through $(\bar x, \bar y)$;
• $SS_{tot} = SS_{reg} + SS_{res}$, and $R^2 = SS_{reg}/SS_{tot} = r^2$;
• the regression of $x$ on $y$ is a different line (slope product $= r^2$).

Multiple linear regression

Stack the data into the design matrix $X$ ($n \times (p+1)$, a column of ones for the intercept):

$$\hat{\boldsymbol\beta} = \arg\min\lVert\mathbf y - X\boldsymbol\beta\rVert^2 = (X^\top X)^{-1}X^\top\mathbf y.$$

Geometry: $\hat{\mathbf y} = X\hat{\boldsymbol\beta} = H\mathbf y$ is the orthogonal projection of $\mathbf y$ onto the column space of $X$, with hat matrix $H = X(X^\top X)^{-1}X^\top$; the residual vector is orthogonal to every column (see projections).

Interpretation: $\hat\beta_j$ is the expected change in $y$ for a one-unit increase in $x_j$ holding the other features fixed. Coefficients from a multiple regression can differ (even in sign) from simple-regression slopes when features are correlated.

Categorical predictors and interactions

• A categorical feature with $k$ levels becomes $k - 1$ dummy (one-hot) variables; using all $k$ with an intercept makes $X$ rank-deficient (the dummy-variable trap).
• An interaction term $x_1x_2$ lets the effect of $x_1$ depend on $x_2$.

Assumptions and diagnostics

For inference (standard errors, t-tests, confidence intervals) the classical assumptions are: linearity, independent errors, homoscedasticity (constant error variance), normally distributed errors, and no perfect multicollinearity. Under the first four (Gauss–Markov without normality), OLS is the best linear unbiased estimator (BLUE); with normal errors it is also the maximum likelihood estimator.

Each coefficient's significance is tested with $t = \hat\beta_j/\text{SE}(\hat\beta_j)$ on $n - p - 1$ degrees of freedom; the error variance is estimated by $\hat\sigma^2 = SS_{res}/(n - p - 1)$.

Fitting by gradient descent

When $p$ is large, solving the normal equations ($O(np^2 + p^3)$) is replaced by gradient descent on the MSE:

$$\nabla_{\boldsymbol\beta}\frac1n\lVert X\boldsymbol\beta - \mathbf y\rVert^2 = \frac2n X^\top(X\boldsymbol\beta - \mathbf y), \qquad \boldsymbol\beta \leftarrow \boldsymbol\beta - \eta\,\frac2nX^\top(X\boldsymbol\beta - \mathbf y).$$

The loss is convex (a quadratic bowl), so gradient descent converges to the global minimum for a small enough $\eta$; feature scaling speeds it up dramatically. Stochastic or mini-batch versions use a subset of rows per step.

Quick revision