Sampling Distributions, the Central Limit Theorem and Confidence Intervals

Parameters, statistics and sampling distributions

A parameter describes the population ($\mu, \sigma^2, p$) and is fixed but unknown. A statistic is computed from a sample ($\bar X, s^2, \hat p$) and varies from sample to sample. The distribution of a statistic over repeated samples is its sampling distribution.

For an i.i.d. sample of size $n$ from a population with mean $\mu$ and variance $\sigma^2$:

$$E[\bar X] = \mu, \qquad \operatorname{Var}(\bar X) = \frac{\sigma^2}{n}, \qquad \text{SE}(\bar X) = \frac{\sigma}{\sqrt n}.$$

Averaging reduces spread by a factor of $\sqrt n$: to halve the standard error you need four times the data.

Law of large numbers

$\bar X_n \to \mu$ as $n \to \infty$ (in probability for the weak law, almost surely for the strong law). This justifies estimating probabilities by relative frequencies and expectations by sample averages (Monte Carlo methods, see approximate inference).

The central limit theorem

If $X_1, \dots, X_n$ are i.i.d. with mean $\mu$ and finite variance $\sigma^2$, then

$$\frac{\bar X_n - \mu}{\sigma/\sqrt n} \xrightarrow{d} N(0, 1), \qquad\text{equivalently}\qquad \sum_{i=1}^n X_i \approx N(n\mu, n\sigma^2).$$

The population can be skewed, discrete, anything with finite variance. In practice $n \ge 30$ is the usual rule of thumb; strongly skewed populations need more, and normal populations need none (the mean is exactly normal).

Point estimation

An estimator $\hat\theta$ is a statistic used to guess $\theta$.

Maximum likelihood estimation picks $\hat\theta = \arg\max_\theta \prod_i f(x_i; \theta)$, usually by maximising the log-likelihood. Standard results: for Bernoulli, $\hat p = \bar x$; for Poisson, $\hat\lambda = \bar x$; for exponential, $\hat\lambda = 1/\bar x$; for normal, $\hat\mu = \bar x$ and $\hat\sigma^2 = \frac1n\sum(x_i - \bar x)^2$ (biased, divides by $n$).

Confidence intervals

A $100(1-\alpha)\%$ confidence interval is a random interval that contains the true parameter with probability $1 - \alpha$ before the data are seen. General shape:

$$\text{estimate} \pm \text{critical value} \times \text{standard error}.$$

Here $z_{\alpha/2}$ is the value with upper-tail area $\alpha/2$: 1.645 (90%), 1.96 (95%), 2.576 (99%). The variance interval is not symmetric about $s^2$ because the chi-squared distribution is skewed.

What controls the width

Width $= 2 \times \text{critical value} \times \sigma/\sqrt n$:

• higher confidence → wider;
• more variable data → wider;
• larger sample → narrower (by $\sqrt n$).

Sample size for a target margin $E$: $n = \left\lceil\left(\dfrac{z_{\alpha/2}\,\sigma}{E}\right)^2\right\rceil$; for a proportion with no prior estimate, use $p = 0.5$: $n = \lceil z^2/(4E^2)\rceil$ (1068 for $\pm 3\%$ at 95%).

Confidence intervals and tests

A two-sided test of $H_0: \mu = \mu_0$ at level $\alpha$ rejects exactly when $\mu_0$ lies outside the $100(1-\alpha)\%$ confidence interval. See hypothesis testing.

Quick revision