Joint Distributions

Most interesting questions involve two quantities at once. Does it rain, and did you bring an umbrella? How tall is a person, and how heavy? Which product did a shopper buy, and how much did they spend? A single distribution describes one random variable on its own — a joint distribution describes two of them together, and crucially it captures how they move with each other.

For two discrete random variables X and Y, the joint probability mass function gives the probability that they land on a particular pair of values at the same time:

p(x, y) = P(X = x \text{ and } Y = y).

Lay every pair out in a grid — one row per value of X, one column per value of Y — and you have the whole joint distribution in a table. Because X and Y must land somewhere, every entry is between 0 and 1 and the entries add up to 1:

\sum_{x}\sum_{y} p(x, y) = 1.

You could write down how often it rains, and separately how often you carry an umbrella — but that throws away the interesting part: whether the umbrella tends to come out on the rainy days. The joint table keeps that link. As we'll see, the two "separate" pictures (the marginals) can be identical for two situations that behave completely differently. The marginals can't tell the whole story — you need the joint.

A worked table: weather and umbrellas

Pick a random day in a rainy town. Let X record the weather (Rain / No rain) and Y record whether you carried an umbrella (Umbrella / No umbrella). A year of record-keeping gives this joint table (each cell is p(x,y)):

	Umbrella	No umbrella
Rain	0.20	0.10
No rain	0.15	0.55

The four numbers add to 0.20 + 0.10 + 0.15 + 0.55 = 1, as a joint PMF must. Read a single cell straight off: the chance of a rainy day with an umbrella is p(\text{Rain}, \text{Umbrella}) = 0.20; the chance of a dry day with no umbrella is 0.55. The interactive grid below rebuilds this table step by step — and fills in the margins.

Marginals: writing the totals in the margins

To get the distribution of X on its own, you no longer care what Y did — so you add up over every value of Y. That collapses each row to a single total:

p_X(x) = \sum_{y} p(x, y), \qquad p_Y(y) = \sum_{x} p(x, y).

These are the marginal distributions. The name is wonderfully literal: you write the row totals down the right margin of the table and the column totals along the bottom margin. For our weather table:

Weather marginal (row sums): P(\text{Rain}) = 0.20 + 0.10 = 0.30 and P(\text{No rain}) = 0.15 + 0.55 = 0.70.
Umbrella marginal (column sums): P(\text{Umbrella}) = 0.20 + 0.15 = 0.35 and P(\text{No umbrella}) = 0.10 + 0.55 = 0.65.

Each marginal is a genuine distribution — its own entries add to 1 (0.30 + 0.70 = 1, and 0.35 + 0.65 = 1). The grand total in the corner is 1 either way you slice it.

If X and Y are continuous (heights, weights, times), the joint PMF becomes a joint density f(x, y) — a surface over the plane whose total volume is 1. Sums turn into integrals, so the marginals come from integrating out the other variable:

f_X(x) = \int_{-\infty}^{\infty} f(x, y)\,dy, \qquad f_Y(y) = \int_{-\infty}^{\infty} f(x, y)\,dx.

The idea is identical — "sum over the other variable" just becomes "integrate over it".

Conditional distributions: zoom into one row

Suppose you already know it's raining. Among rainy days, how often do you actually carry the umbrella? That's a conditional distribution — you throw away every day that isn't rainy, keep the Rain row, and rescale it so it adds to 1 again. This is exactly conditional probability applied cell by cell:

P(Y = y \mid X = x) = \frac{p(x, y)}{p_X(x)}.

Divide each entry of the Rain row by that row's marginal p_X(\text{Rain}) = 0.30:

P(\text{Umbrella} \mid \text{Rain}) = \frac{0.20}{0.30} \approx 0.67, \qquad P(\text{No umbrella} \mid \text{Rain}) = \frac{0.10}{0.30} \approx 0.33.

Those two rescaled numbers add to 1 — a fresh distribution over the umbrella, given that it rained. So on a rainy day you carry the umbrella about two times in three. Compare that with the unconditional P(\text{Umbrella}) = 0.35: knowing it rains changes the umbrella's distribution. That change is the fingerprint of dependence.

Independence: does every cell factor?

Two random variables are independent when knowing one tells you nothing new about the other. In table terms that has a beautifully clean test: the joint must equal the product of the marginals, in every single cell:

X and Y are independent \iff p(x, y) = p_X(x)\,p_Y(y) for every pair (x, y).
Equivalently, every conditional equals the corresponding marginal: P(Y = y \mid X = x) = p_Y(y) — conditioning changes nothing.
One cell that fails to factor is enough to make them dependent.

Test the weather table on its rainy-umbrella cell. The product of the marginals predicts

p_X(\text{Rain})\,p_Y(\text{Umbrella}) = 0.30 \times 0.35 = 0.105,

but the actual joint entry is 0.20. Since 0.20 \neq 0.105, this cell fails to factor — so weather and umbrella are not independent. Good: umbrellas come out precisely because it rains.

Same margins, different stories

Here is the punchline that makes joint distributions worth their own lesson. Take two variables X and Y that each take the values 0 and 1, both with marginal (0.5,\ 0.5). Two completely different joint tables produce those same marginals:

Independent	Y=0	Y=1
X=0	0.25	0.25
X=1	0.25	0.25

Dependent	Y=0	Y=1
X=0	0.40	0.10
X=1	0.10	0.40

Check the margins of both: every row and every column sums to 0.5. Identical marginals. Yet in the first table every cell factors (0.25 = 0.5 \times 0.5) — X and Y are independent — while in the second the diagonal is heavy (0.40 \neq 0.5 \times 0.5 = 0.25), so the two variables are strongly linked: they tend to agree. Same marginals, opposite behaviour. The marginals simply do not determine the joint.

Squeezing every last drop of information out of a joint distribution — its correlations, and eventually a single function that encodes all of its moments — leads on to moment generating functions.

You can always recover the marginals from the joint — just sum over the other variable. But you can not recover the joint from the marginals. Two tables with identical marginals can be independent or strongly dependent (the pair of tables above).
Independence must hold for every cell, p(x,y) = p_X(x)\,p_Y(y) throughout. One cell that happens to factor is not enough — check them all (or find a single cell that fails, which settles it).
Don't confuse the joint p(x,y)=P(X=x\text{ and }Y=y) with the conditional P(Y=y\mid X=x). The conditional is the joint divided by a marginal — a single rescaled row, not the raw cell.