Gradient Descent

Rolling downhill

An interactive lesson with 7 steps, about 6 minutes: you predict, drag and solve, with instant feedback on every answer. It runs in your browser and needs JavaScript.

What you’ll learn

• A loss function measures how wrong a network is.
• Gradient descent: $w \leftarrow w - \eta\, \partial L / \partial w$.
• Too small a learning rate is slow; too large overshoots or diverges.

Steps

1. Measuring wrongness
2. Choose a learning rate
3. One small step
4. Too big a step
5. A ball with momentum
6. Stuck in a valley?
7. Mini-batches