Sample Spaces, Axioms, Conditional and Joint Probability

Experiments, sample spaces and events

A random experiment has an outcome that cannot be predicted with certainty. Its sample space $S$ (or $\Omega$) is the set of all possible outcomes; an event is a subset of $S$.

Events combine with set operations: $A \cup B$ (A or B), $A \cap B$ (A and B), $A^c$ (not A), $A \setminus B = A \cap B^c$. De Morgan's laws: $(A\cup B)^c = A^c \cap B^c$ and $(A \cap B)^c = A^c \cup B^c$.

Events are mutually exclusive (disjoint) if $A \cap B = \emptyset$, and exhaustive if $A_1 \cup \dots \cup A_n = S$. A collection that is both is a partition of $S$.

The axioms of probability (Kolmogorov)

A probability measure assigns a number $P(A)$ to each event so that

1. $P(A) \ge 0$ for every event $A$;
2. $P(S) = 1$;
3. for any countable sequence of pairwise disjoint events, $P\left(\bigcup_i A_i\right) = \sum_i P(A_i)$.

Everything else is a theorem:

Assigning probabilities

• Classical (equally likely outcomes): $P(A) = |A| / |S|$ — counting, see Counting.
• Relative frequency: $P(A) = \lim_{n\to\infty} n_A / n$.
• Geometric: for a point chosen uniformly in a region, $P(A) = \text{area}(A) / \text{area}(S)$.

Conditional probability

The probability of $A$ given that $B$ has occurred (with $P(B) > 0$):

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}.$$

Conditioning shrinks the sample space to $B$ and rescales. $P(\cdot \mid B)$ is itself a valid probability measure, so all the rules above hold conditionally, e.g. $P(A^c \mid B) = 1 - P(A\mid B)$.

Multiplication (chain) rule

$$P(A \cap B) = P(B)\,P(A\mid B) = P(A)\,P(B \mid A)$$ $$P(A_1 \cap A_2 \cap \dots \cap A_n) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1\cap A_2) \cdots P(A_n \mid A_1 \cap\dots\cap A_{n-1}).$$

Total probability

If $B_1, \dots, B_n$ partition $S$:

$$P(A) = \sum_{i=1}^n P(A \mid B_i)\,P(B_i).$$

This is the denominator of Bayes' theorem — see Bayes' theorem.

Joint, marginal and conditional probability

For two discrete variables the joint distribution is a table; marginals are row/column sums; conditionals are a row (or column) divided by its sum.

• Marginal: $P(Y=1) = 0.30 + 0.40 = 0.70$ (sum over the other variable — marginalisation).
• Conditional: $P(Y = 1 \mid X = 0) = 0.30 / 0.40 = 0.75$.
• Independence check: $P(X=0)P(Y=0) = 0.12 \ne 0.10$, so $X$ and $Y$ are dependent.

Independence

Events $A$ and $B$ are independent if

$$P(A \cap B) = P(A)\,P(B) \quad\Longleftrightarrow\quad P(A\mid B) = P(A)\ \ (\text{when } P(B) > 0).$$

Knowing $B$ happened tells you nothing about $A$. If $A, B$ are independent, so are $A$ and $B^c$, $A^c$ and $B$, and $A^c$ and $B^c$.

Pairwise vs mutual independence

$A, B, C$ are mutually independent if all of these hold:

$$P(A\cap B) = P(A)P(B),\ P(A\cap C) = P(A)P(C),\ P(B\cap C) = P(B)P(C),\ P(A\cap B\cap C) = P(A)P(B)P(C).$$

Pairwise independence (the first three) does not imply the fourth — the two-coin example in the practice questions is the standard counterexample. For $n$ events, mutual independence needs $2^n - n - 1$ conditions.

Mutually exclusive vs independent

Independent trials

For independent events, "at least one occurs" is easiest via the complement:

$$P(\text{at least one of } A_1,\dots,A_n) = 1 - \prod_{i=1}^n \bigl(1 - P(A_i)\bigr).$$

Conditional independence

$A$ and $B$ are conditionally independent given $C$ if $P(A\cap B \mid C) = P(A\mid C)\,P(B \mid C)$. Neither independence nor conditional independence implies the other. This idea underlies naive Bayes and Bayesian networks — see Bayesian networks.

Quick revision