Multiple Comparisons

Test enough, and something will be “significant”

An interactive lesson with 6 steps, about 5 minutes: you predict, drag and solve, with instant feedback on every answer. It runs in your browser and needs JavaScript.

What you’ll learn

• With $k$ independent tests at $\alpha = 0.05$: $P(\text{false positive}) = 1 - 0.95^k$.
• Bonferroni: use $\alpha / k$ for each test.
• Hidden comparisons (p-hacking) make findings look stronger than they are.

Steps

1. The jelly bean problem
2. Bonferroni
3. How many findings are true?
4. Count the discoveries
5. After the correction
6. Torturing the data