Descriptive Statistics — Mean, Median, Mode, Spread, Covariance and Correlation

Descriptive statistics summarise a data set with a few numbers: where it is centred, how spread out it is, its shape, and how two variables move together. These formulas reappear in random variables (as expectations), in hypothesis testing (sample mean and variance), in PCA (the covariance matrix) and in regression (correlation and least squares).

Measures of central tendency

Key property of the mean: deviations sum to zero, $\sum (x_i - \bar x) = 0$, and $\bar x$ minimises $\sum (x_i - c)^2$. The median minimises $\sum |x_i - c|$.

Grouped data

For a frequency table with class midpoints $m_i$ and frequencies $f_i$: $\bar x = \sum f_i m_i / \sum f_i$.

• Median: $L + \dfrac{n/2 - F}{f}\,h$ where $L$ is the lower boundary of the median class, $F$ the cumulative frequency before it, $f$ its frequency and $h$ the class width.
• Mode: $L + \dfrac{f_1 - f_0}{2f_1 - f_0 - f_2}\,h$ (modal class $f_1$, neighbours $f_0, f_2$).
• Empirical relation for moderately skewed data: $\text{mode} \approx 3\,\text{median} - 2\,\text{mean}$.

Measures of spread

Shifting and scaling

If $y_i = a x_i + b$: $\bar y = a\bar x + b$, $\operatorname{median}_y = a\operatorname{median}_x + b$, $s_y = |a|\,s_x$, $s_y^2 = a^2 s_x^2$. Shifting moves location but never spread.

Standardisation: $z_i = (x_i - \bar x)/s$ has mean 0 and standard deviation 1.

Quantiles, box plots and skewness

The $p$-th quantile has a fraction $p$ of the data below it; $Q_1, Q_2 = \text{median}, Q_3$ are the quartiles. A box plot draws $Q_1$–$Q_3$ as a box with the median inside, whiskers to the most extreme points within $1.5\,\text{IQR}$, and flags points beyond as outliers.

Moment coefficient of skewness: $\gamma_1 = \frac{E[(X-\mu)^3]}{\sigma^3}$ (0 for symmetric distributions). Kurtosis $\frac{E[(X-\mu)^4]}{\sigma^4}$ is 3 for the normal distribution.

Covariance

$$\operatorname{Cov}(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - E[X]E[Y].$$

For data: $s_{xy} = \frac{1}{n-1}\sum (x_i - \bar x)(y_i - \bar y)$. Positive covariance means the variables tend to be above their means together.

Properties:

• $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$; $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$.
• Bilinear: $\operatorname{Cov}(aX + b, cY + d) = ac\,\operatorname{Cov}(X, Y)$.
• $\operatorname{Var}(X \pm Y) = \operatorname{Var} X + \operatorname{Var}Y \pm 2\operatorname{Cov}(X,Y)$.
• Independent $\Rightarrow$ $\operatorname{Cov} = 0$ (the converse is false).

Covariance depends on units, which is why we normalise it.

Pearson correlation

$$\rho_{XY} = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\sigma_Y}, \qquad r = \frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum (x_i - \bar x)^2 \sum (y_i - \bar y)^2}}.$$

Spearman's rank correlation is Pearson's $r$ applied to ranks, capturing monotonic (not just linear) relationships: $r_s = 1 - \frac{6\sum d_i^2}{n(n^2-1)}$ when there are no ties.

The covariance matrix

For a random vector $\mathbf X = (X_1, \dots, X_p)$, $\Sigma_{ij} = \operatorname{Cov}(X_i, X_j)$. $\Sigma$ is symmetric and positive semi-definite, and $\operatorname{Var}(\mathbf a^\top \mathbf X) = \mathbf a^\top \Sigma\, \mathbf a$. This is the matrix PCA diagonalises — see Principal component analysis.

Quick revision