Bayes' Theorem: Prior, Likelihood and Posterior

Imagine a doctor with a test that's 99% accurate for a rare disease that only 1 in 10,000 people have. A patient tests positive — how worried should they be? Most people's gut answer ('99% likely to have it') is wrong, and Bayes' theorem is the tool that gets you the right answer. It's arguably the single most important formula in this whole prerequisites unit, because it's the mathematical backbone of how you update a belief when new evidence arrives.

Bayes' theorem has three named ingredients. The prior, P(A), is what you believed BEFORE seeing any evidence — here, P(disease) = 1/10,000, since that's the rate in the general population before we know anything about this specific patient. The likelihood, P(evidence | A), is how probable the evidence is IF your belief were true — P(positive test | disease) = 0.99, since the test catches 99% of true cases. The posterior, P(A | evidence), is what you actually want: your UPDATED belief after seeing the evidence — P(disease | positive test).

The formula ties them together: P(A | evidence) = P(evidence | A) x P(A) / P(evidence). Notice this is just the conditional probability formula from the previous subtopic, rearranged — Bayes' theorem is really just 'solve P(A and B) = P(A) x P(B|A) for the OTHER conditional direction.' It flips a question you're given (how likely is the evidence, assuming the belief) into the question you actually want answered (how likely is the belief, given the evidence) — these are NOT the same number, and confusing them is exactly the mistake most people make on the doctor's-test puzzle above. Working through the math (see code below), even with a 99%-accurate test, a positive result on a disease this rare still leaves under a 1% chance the patient actually has it — because false positives from the huge healthy population outnumber the true positives from the tiny sick population.

Bayes' theorem matters for LLMs because it's the formal justification for treating a model's output as a BELIEF that gets updated by evidence: a language model's 'prior' over the next token gets updated ('posterior') by the evidence of the prompt/context so far, and Bayesian thinking underlies concepts like calibration (does the model's confidence match its actual accuracy?) that show up throughout LLM evaluation.

# The classic rare-disease puzzle, worked with Bayes' theorem --
# showing why a 99%-accurate test can still mean <1% posterior risk.

def bayes_posterior(prior, likelihood, false_positive_rate):
    """P(disease | positive test) via Bayes' theorem.
    P(evidence) is expanded as: P(disease)*P(pos|disease) +
                                P(no disease)*P(pos|no disease)
    """
    p_disease = prior
    p_no_disease = 1 - prior
    p_evidence = (p_disease * likelihood) + (p_no_disease * false_positive_rate)
    posterior = (likelihood * p_disease) / p_evidence
    return posterior

prior = 1 / 10_000          # P(disease) before any test -- 0.01%
likelihood = 0.99           # P(positive test | disease)  -- 99% sensitivity
false_positive_rate = 0.01  # P(positive test | no disease) -- 1% false-positive rate

posterior = bayes_posterior(prior, likelihood, false_positive_rate)
print(f"Prior P(disease):             {prior:.4%}")
print(f"Posterior P(disease | +test): {posterior:.4%}")
# Posterior is under 1% -- the rarity of the disease dominates the
# test's accuracy, which is the whole counter-intuitive point