Information & Entropy

Shannon’s measure of surprise

An interactive lesson with 6 steps, about 5 minutes: you predict, drag and solve, with instant feedback on every answer. It runs in your browser and needs JavaScript.

What you’ll learn

• An event with probability $p$ carries $\log_2(1/p)$ bits of information.
• Entropy is the average information. It is highest when outcomes are equally likely.
• Predictable data has low entropy, which is why it compresses well.

Steps

1. Twenty questions
2. Information is surprise
3. One card
4. Entropy
5. A fair die
6. Why compression works