Probability Fundamentals: Events, Distributions and Joint/Conditional/Marginal Probability

Flip a coin and you don't know if it'll land heads or tails — but you DO know something useful: heads happens about half the time. Probability is just a number between 0 and 1 that captures that kind of knowledge. 0 means 'never happens,' 1 means 'always happens,' and 0.5 means 'happens about half the time.' An event is simply the thing you're asking about — 'the coin lands heads,' 'the next word is happy,' 'it rains tomorrow.'

A probability distribution is just a list of every possible outcome paired with its probability, where all the probabilities add up to 1. Roll a fair six-sided die and the distribution is dead simple: each of the 6 faces gets probability 1/6. A language model's vocabulary works the same way at every step: it assigns a probability to every possible next word, and all of those probabilities sum to 1.

Once you have two things that can happen, three natural questions come up, and probability has a name for each. Joint probability asks: what's the chance BOTH things happen? P(rain AND traffic) is the joint probability of rain and traffic both occurring today. Marginal probability asks about just ONE thing, ignoring the other entirely — P(rain), no matter what traffic does. You get it by adding up all the joint probabilities that involve rain, across every possible traffic outcome. Conditional probability asks: GIVEN that one thing happened, what's the chance of the other? P(traffic | rain) — the probability of traffic, given that we already know it's raining — is usually higher than the plain P(traffic), because rain and traffic are related.

These three connect through one simple, memorable formula: P(A and B) = P(A) x P(B | A). The joint probability is the marginal probability of the first thing, times the conditional probability of the second thing GIVEN the first. This formula is the seed that Bayes' theorem (next subtopic) grows out of, and it's exactly how a language model computes the probability of an entire sentence: multiply the conditional probability of each word given all the words before it.

# Simulating a distribution, and computing joint / marginal /
# conditional probability from raw counts -- no library needed.

# Toy dataset: 100 days, recording (rain?, traffic?)
days = (
    [("rain", "traffic")] * 30 +
    [("rain", "no_traffic")] * 10 +
    [("no_rain", "traffic")] * 20 +
    [("no_rain", "no_traffic")] * 40
)
total = len(days)

def joint_prob(a, b):
    return sum(1 for x, y in days if x == a and y == b) / total

def marginal_prob(a):
    """Ignore the other variable entirely -- match `a` in either
    position of the (rain_status, traffic_status) pair."""
    return sum(1 for pair in days if a in pair) / total

def conditional_prob(b_given, a):
    """P(b_given | a) = P(a and b_given) / P(a)"""
    return joint_prob(a, b_given) / marginal_prob(a)

print(f"P(rain)                = {marginal_prob('rain'):.2f}")
print(f"P(rain AND traffic)     = {joint_prob('rain', 'traffic'):.2f}")
print(f"P(traffic | rain)       = {conditional_prob('traffic', 'rain'):.2f}")
print(f"P(traffic)  (marginal)  = {marginal_prob('traffic'):.2f}")
# Traffic is much more likely GIVEN rain (0.75) than in general (0.50)