Linear Discriminant Analysis

LDA as a classifier

LDA is a generative model: it models how each class generates its features, then classifies with Bayes' theorem.

• Prior: $P(y = k) = \pi_k$ (estimated by class proportions).
• Class-conditional density: $\mathbf x \mid y = k \sim N(\boldsymbol\mu_k, \Sigma)$ — Gaussian with a class-specific mean and a shared covariance.

Classify by the largest posterior, i.e. the largest discriminant function (log of $\pi_k\,p(\mathbf x\mid k)$ with common terms dropped):

$$\delta_k(\mathbf x) = \mathbf x^\top\Sigma^{-1}\boldsymbol\mu_k - \frac12\boldsymbol\mu_k^\top\Sigma^{-1}\boldsymbol\mu_k + \ln\pi_k.$$

$\delta_k$ is linear in $\mathbf x$, so the boundary $\{\delta_k = \delta_j\}$ between any two classes is a hyperplane.

Estimation: $\hat\pi_k = n_k/n$, $\hat{\boldsymbol\mu}_k$ = class sample mean, $\hat\Sigma$ = pooled within-class covariance $\frac{1}{n - K}\sum_k\sum_{i\in k}(\mathbf x_i - \hat{\boldsymbol\mu}_k)(\mathbf x_i - \hat{\boldsymbol\mu}_k)^\top$.

The one-dimensional case

With means $\mu_1, \mu_2$, common variance $\sigma^2$ and priors $\pi_1, \pi_2$, the boundary is

$$x^* = \frac{\mu_1 + \mu_2}{2} + \frac{\sigma^2}{\mu_2 - \mu_1}\ln\frac{\pi_1}{\pi_2}.$$

Equal priors give the midpoint; a larger prior for class 1 pushes the boundary towards class 2.

Quadratic discriminant analysis (QDA)

If each class has its own covariance $\Sigma_k$, the quadratic terms no longer cancel:

$$\delta_k(\mathbf x) = -\frac12\ln\lvert\Sigma_k\rvert - \frac12(\mathbf x - \boldsymbol\mu_k)^\top\Sigma_k^{-1}(\mathbf x - \boldsymbol\mu_k) + \ln\pi_k,$$

giving quadratic boundaries. QDA is more flexible (lower bias) but estimates $K\frac{d(d+1)}{2}$ covariance parameters (higher variance). Gaussian naive Bayes is QDA with diagonal covariances.

Fisher's LDA: supervised dimensionality reduction

Find a projection $z = \mathbf w^\top\mathbf x$ that separates the class means while keeping each class compact. For two classes, maximise Fisher's criterion

$$J(\mathbf w) = \frac{(\mathbf w^\top(\mathbf m_1 - \mathbf m_2))^2}{\mathbf w^\top S_W\mathbf w} = \frac{\mathbf w^\top S_B\mathbf w}{\mathbf w^\top S_W\mathbf w},$$

with within-class scatter $S_W = \sum_k\sum_{i\in k}(\mathbf x_i - \mathbf m_k)(\mathbf x_i - \mathbf m_k)^\top$ and between-class scatter $S_B = (\mathbf m_1 - \mathbf m_2)(\mathbf m_1 - \mathbf m_2)^\top$. The solution is

$$\mathbf w \propto S_W^{-1}(\mathbf m_1 - \mathbf m_2).$$

For $K$ classes, the directions are the top eigenvectors of $S_W^{-1}S_B$, at most $K - 1$ of them. Under LDA's Gaussian assumptions, classifying in this projected space is equivalent to the LDA classifier.

LDA vs related methods

Quick revision