Random Variables, Expectation, Joint and Conditional Distributions

What a random variable is

A random variable (RV) is a function $X: S \to \mathbb R$ that assigns a number to each outcome. "$X = 3$" is shorthand for the event $\{s \in S : X(s) = 3\}$.

• Discrete RVs take countably many values (counts, indicators).
• Continuous RVs take values in intervals and have $P(X = x) = 0$ for every single point.

PMF, PDF and CDF

CDF properties: non-decreasing, right-continuous, $F(-\infty) = 0$, $F(\infty) = 1$, and $P(a < X \le b) = F(b) - F(a)$.

Expectation

$$E[X] = \sum_x x\,p(x) \qquad\text{or}\qquad E[X] = \int_{-\infty}^\infty x\,f(x)\,dx.$$

Law of the unconscious statistician (LOTUS): $E[g(X)] = \sum_x g(x)p(x)$ or $\int g(x)f(x)\,dx$ — no need to find the distribution of $g(X)$.

Linearity holds always, even for dependent variables: $E[aX + bY + c] = aE[X] + bE[Y] + c$.

For a non-negative integer RV, $E[X] = \sum_{k \ge 1} P(X \ge k)$; for a non-negative continuous RV, $E[X] = \int_0^\infty P(X > x)\,dx$.

Variance and moments

$$\operatorname{Var}(X) = E[(X - \mu)^2] = E[X^2] - (E[X])^2, \qquad \operatorname{Var}(aX + b) = a^2\operatorname{Var}(X).$$

The $k$-th moment is $E[X^k]$; the $k$-th central moment is $E[(X-\mu)^k]$. The moment generating function $M_X(t) = E[e^{tX}]$ gives moments by differentiation, $E[X^k] = M_X^{(k)}(0)$, and the MGF of a sum of independent RVs is the product of their MGFs.

Functions of a random variable

For continuous $X$ and a strictly monotone $g$, $Y = g(X)$ has density

$$f_Y(y) = f_X\bigl(g^{-1}(y)\bigr)\left|\frac{d}{dy}g^{-1}(y)\right|.$$

The safest general method is the CDF method: compute $F_Y(y) = P(g(X) \le y)$ directly, then differentiate.

Inverse transform sampling: if $U \sim U(0,1)$ and $F$ is a CDF, then $F^{-1}(U)$ has CDF $F$. For example $-\ln(U)/\lambda \sim \text{Exp}(\lambda)$. This is how samplers generate arbitrary distributions — see approximate inference.

Joint distributions

For two RVs, the joint PMF $p(x, y) = P(X = x, Y = y)$ or joint PDF $f(x, y)$ with $P((X,Y) \in A) = \iint_A f(x,y)\,dx\,dy$.

The conditional PDF $f_{Y\mid X}(y \mid x)$ is a genuine density in $y$ for each fixed $x$ (it integrates to 1).

For independent $X, Y$: $E[XY] = E[X]E[Y]$, $\operatorname{Var}(X+Y) = \operatorname{Var}X + \operatorname{Var}Y$, and $E[g(X)h(Y)] = E[g(X)]E[h(Y)]$.

Sums of independent random variables

The PMF/PDF of $Z = X + Y$ is a convolution: $f_Z(z) = \int f_X(x)f_Y(z - x)\,dx$. Useful closed forms: sum of independent Poissons is Poisson ($\lambda_1 + \lambda_2$); sum of independent normals is normal (means add, variances add); sum of $n$ i.i.d. Bernoulli($p$) is Binomial($n, p$).

Conditional expectation and variance

$E[Y \mid X = x] = \sum_y y\,p(y\mid x)$ or $\int y\,f(y \mid x)\,dy$ is a number for each $x$; $E[Y\mid X]$ is a random variable (a function of $X$).

Law of total expectation (tower property):

$$E[Y] = E\bigl[E[Y \mid X]\bigr].$$

Law of total variance (Eve's law):

$$\operatorname{Var}(Y) = \underbrace{E[\operatorname{Var}(Y\mid X)]}_{\text{within-group}} + \underbrace{\operatorname{Var}(E[Y \mid X])}_{\text{between-group}}.$$

$E[Y\mid X]$ is the best predictor of $Y$ from $X$ in mean squared error — the quantity regression tries to estimate.

Inequalities

Quick revision