Total Probability and Bayes' Theorem

The law of total probability, Bayes' theorem with priors, likelihoods and posteriors, tree diagrams, the odds form and likelihood ratios, the base-rate fallacy, sequential updating, and the MAP/ML decision rules that machine learning builds on.

Section 19 practice questions

The law of total probability

Let $B_1, \dots, B_n$ be a partition of the sample space (disjoint, exhaustive, each with positive probability). Then for any event $A$ ,

P(A) = \sum_{i=1}^n P(A \cap B_i) = \sum_{i=1}^n P(A \mid B_i)\,P(B_i).

Think of it as "average the conditional probabilities, weighted by how likely each scenario is". The continuous version replaces the sum by an integral: $P(A) = \int P(A \mid X = x) f_X(x)\,dx$ .

Bayes' theorem

Combining the definition of conditional probability with total probability:

P(B_j \mid A) = \frac{P(A \mid B_j)\,P(B_j)}{\sum_{i=1}^n P(A\mid B_i)\,P(B_i)}.

Term	Name	Meaning
$P(B_j)$	prior	belief in hypothesis $B_j$ before seeing evidence
$P(A \mid B_j)$	likelihood	how probable the evidence is if $B_j$ is true
$P(A)$	evidence / marginal likelihood	normalising constant
$P(B_j\mid A)$	posterior	updated belief after seeing $A$

\text{posterior} \;\propto\; \text{likelihood} \times \text{prior}.

Multiply along branches for joint probabilities; P(D | +) = 0.0095 / (0.0095 + 0.0495) ≈ 0.16.

The base-rate fallacy

When the prior $P(D)$ is tiny, even an accurate test produces mostly false positives. In the tree above, the 0.0495 false-positive mass is five times the 0.0095 true-positive mass. Medical screening, fraud detection and spam filtering all face this: the precision of a detector depends on the base rate, while its recall (sensitivity) does not.

Test quantity	Definition
Sensitivity (recall, TPR)	$P(+\mid D)$
Specificity (TNR)	$P(-\mid D^c)$
False-positive rate	$1 - \text{specificity} = P(+\mid D^c)$
Positive predictive value (precision)	$P(D\mid +)$ — needs the prior

Odds form and likelihood ratios

Odds $O(H) = P(H)/P(H^c)$ . Bayes' theorem becomes

\underbrace{\frac{P(H\mid E)}{P(H^c\mid E)}}_{\text{posterior odds}} = \underbrace{\frac{P(E\mid H)}{P(E\mid H^c)}}_{\text{likelihood ratio}} \times \underbrace{\frac{P(H)}{P(H^c)}}_{\text{prior odds}}.

With conditionally independent pieces of evidence $E_1, E_2, \dots$ , the likelihood ratios multiply — fast for repeated tests. Convert back with $P = O/(1+O)$ .

Sequential updating

Today's posterior is tomorrow's prior. Updating on $E_1$ then $E_2$ gives the same result as updating on both at once, provided you use $P(E_2 \mid H, E_1)$ in the second step (which equals $P(E_2\mid H)$ under conditional independence).

Decision rules: MAP and ML

To pick one hypothesis after seeing evidence $E$ :

Maximum a posteriori (MAP): choose $\arg\max_H P(H\mid E) = \arg\max_H P(E \mid H)\,P(H)$ (the evidence $P(E)$ does not depend on $H$ ).
Maximum likelihood (ML): choose $\arg\max_H P(E\mid H)$ — MAP with a uniform prior.

The Bayes classifier (MAP) minimises the probability of misclassification. Naive Bayes and LDA are MAP classifiers with particular likelihood models.

Quick revision

Practice questions

GATE-style questions: 2 one-mark and 7 two-mark. Wrong MCQs cost ⅓ of their marks in the real exam; MSQ and NAT have no negative marking. Attempt each before opening the solution.

Q1NAT2 marks

A disease affects 1% of a population. A test detects the disease with probability 0.95 when it is present and gives a false positive with probability 0.05 when it is absent. A randomly chosen person tests positive. The probability that the person has the disease is ______ (round to two decimals).

Q2NAT2 marks

Machines A, B and C produce 50%, 30% and 20% of a factory's output, with defect rates 2%, 3% and 5% respectively. An item chosen at random is defective. The probability that it was produced by machine A is ______ (round to two decimals).

Q3MCQ1 mark

In Bayes' theorem $P(H \mid E) = \dfrac{P(E\mid H)\,P(H)}{P(E)}$ , the term $P(E \mid H)$ is called the

Q4NAT2 marks

Urn 1 contains 3 red and 2 blue balls; urn 2 contains 1 red and 4 blue balls. A fair coin decides which urn to draw from, and one ball is drawn. It is red. The probability that it came from urn 1 is ______.

Q5NAT2 marks

In the Monty Hall game, a car is behind one of three doors. You pick door 1. The host, who knows where the car is and always opens a goat door you did not pick (choosing at random when he has a choice), opens door 3. The probability that you win the car by switching to door 2 is ______ (round to two decimals).

Q6MCQ2 marks

A disease has prevalence 1%; a test has sensitivity 0.95 and false-positive rate 0.05. A person tests positive on two independent tests (conditionally independent given disease status). The probability of disease is closest to

Q7NAT1 mark

If $P(A) = 0.3$ , $P(B) = 0.6$ and $P(B \mid A) = 0.8$ , then $P(A \mid B)$ is ______.

Q8MSQ2 marks

For events $A, B$ with $0 < P(A) < 1$ and $0 < P(B) < 1$ , which of the following always hold?

One or more options are correct.

Q9NAT2 marks

In a multiple-choice exam with 4 options per question, a student knows the answer with probability 0.6 and otherwise guesses uniformly at random. Given that the student answered a question correctly, the probability that the student actually knew the answer is ______ (round to two decimals).