A/B Testing and Experimental Design

A product team argues for a week about whether a green "Buy" button converts better than the blue one. Everyone has an opinion; nobody has evidence. Then they run an A/B test: half the visitors, chosen at random, see blue (the control); the other half see green (the treatment). A week later the numbers settle the argument that a month of meetings could not. This is hypothesis testing wearing a hard hat — the same machinery of nulls, p-values and errors, applied to a live decision with money on it.

An A/B test is just the humblest form of the randomised controlled experiment, the gold standard for breaking confounding. Randomisation is the whole trick: it severs every back-door path at once, so a difference in outcomes must be caused by the change and not by some lurking third variable.

The anatomy of a test

Every well-run experiment nails down the same handful of ingredients before a single visitor is bucketed. Reveal them one at a time.

Notice the shape: a single pool of users, split at random into two arms that are identical except for the one change under test. That "except for one thing" is what lets us attribute the difference in the metric to the change itself. Everything else — time of day, device mix, marketing campaigns, the weather — is balanced across both arms by the randomisation, in expectation.

The null hypothesis H_0 is "no difference": the treatment's true effect on the metric is exactly zero.
The significance level \alpha is the false-positive rate we tolerate — the chance of declaring a winner when H_0 is actually true (a Type I error). Convention is \alpha = 0.05.
The power 1 - \beta is the chance of detecting a real effect when one truly exists — one minus the Type II (false-negative) rate. Convention is 0.8.

Power, effect size and sample size

The single most important number to fix before you start is the sample size — and it is not a guess. It is determined by three choices you make up front: how small an effect you care about, how sure you want to be of catching it, and how much noise your metric carries. These pull against each other in a fixed relationship:

n \;\propto\; \frac{\sigma^2}{(\text{MDE})^2}\,\big(z_{1-\alpha/2} + z_{1-\beta}\big)^2

The minimum detectable effect (MDE) is the smallest lift you would bother to act on — say, a 1% increase in conversions. Read the formula and the trade-offs fall out: to detect a smaller effect (halve the MDE) you need four times the users, because MDE is squared in the denominator. Demanding more power or a stricter \alpha also costs more users, through the (z + z)^2 term. A noisier metric (bigger \sigma^2) costs more too. Running a power analysis to pick n before launch is what stops you from running a test that never had a chance of detecting the effect you hoped for — an "underpowered" test that wastes weeks to conclude nothing.

A worked sizing

Suppose your page converts at 4%, and you would act on a lift to 5% (a one-point, 25\% relative MDE). At \alpha = 0.05 and 80\% power, a standard proportion calculation asks for roughly 7,000 users per arm. Halve your ambition — insist on detecting only a half-point lift to 4.5% — and the requirement balloons to around 28,000 per arm. The lesson is visceral: chasing tiny effects is enormously expensive, and deciding what is worth detecting is a business decision that must happen before, not after, the data arrives.

It is agonising to launch a test and wait two weeks without looking. So people peek: they check the p-value daily and stop the moment it dips below 0.05. This optional stopping quietly wrecks the guarantee. Each peek is another chance to cross the line by luck, so a test designed for a 5% false-positive rate can leak 20–30% false positives if you peek often enough and stop at the first significant reading. The random walk of the p-value will eventually dip below 0.05 by chance even when nothing is happening — and an eager analyst pounces on exactly that moment. The honest fixes: commit to your sample size in advance and only look once, or use methods built for continuous monitoring — sequential testing or always-valid confidence intervals — that spend your \alpha budget across all the peeks.

Peeking is the famous trap, but it has cousins. Multiple testing: try twenty variants (or slice one result across twenty segments) at \alpha = 0.05 and you expect one "winner" by pure chance — correct with a Bonferroni or false-discovery-rate adjustment. Novelty effect: a flashy new feature gets clicked because it is new, and the lift evaporates once the novelty wears off — so run long enough to see steady-state behaviour. Sample-ratio mismatch: if your 50/50 split arrives as 48/52, the randomisation is broken and the whole result is suspect. Not accounting for weekly cycles: a test run only on a weekend measures weekend users, not your population. Each of these turns a rigorous-looking experiment into a confident wrong answer.

Why a data scientist runs experiments

Observational data tells you what did happen; an experiment tells you what will happen if you make a change — the causal question that every product decision secretly asks. Randomisation is the cheapest, most reliable causal-inference tool ever invented, which is why mature tech companies run thousands of A/B tests at once. But the guarantee is fragile: it holds only if you fix your hypothesis and sample size in advance, randomise cleanly, and resist the temptation to stop early or to keep slicing until something looks significant. Design first, then look once.

Learn this on Kaggle

Kaggle doesn't have a dedicated A/B-testing course, but its Kaggle Learn hub covers the underlying statistics, and the community hosts a wealth of hands-on A/B-testing notebooks — search Kaggle for "A/B testing" to find worked experiments that compute power, sample size and significance on real conversion datasets in Python.