A/B Testing

Experiments at scale

An interactive lesson with 7 steps, about 6 minutes: you predict, drag and solve, with instant feedback on every answer. It runs in your browser and needs JavaScript.

What you’ll learn

• Random assignment makes the groups comparable, so differences can be attributed to the change.
• Standard error of a rate: $\sqrt{p(1-p)/n}$. It shrinks like $1/\sqrt n$.
• Small effects need large samples to tell them apart from noise.

Steps

1. Randomised experiments
2. How precise?
3. Is B better?
4. How many visitors?
5. A rule of thumb
6. No peeking
7. Peeking at twenty tests