Lesson 14 of 16 · Learning by Descent
Learning Rate
See how the learning rate controls gradient descent: slow progress when it is too small, zigzags and divergence when it is too big, the maths of when it converges, narrow valleys, learning rate schedules and how to pick a good value.
- 8 min read
- Intermediate
- Reviewed
- What happens when the learning rate is too small, too big or just right
- Why too big a step can make the loss explode
- How the curvature of the loss limits the learning rate
- Why narrow valleys cause zigzagging
- How learning rate schedules and simple searches find a good value
The most important setting
The learning rate η decides how far each gradient descent step goes. It is a hyperparameter: something you choose, not something the model learns.
Three behaviours
Try it on f(x) = x², starting at x = 4. Slide the learning rate from small to large:
| Learning rate | What happens |
|---|---|
| too small | every step is safe but tiny, so training takes forever |
| just right | fast, steady progress to the minimum |
| a bit too big | overshoots, bouncing from side to side but still settling |
| far too big | each step lands higher than the last: the loss diverges to infinity |
Why too big explodes
For f(x) = x² the slope is 2x, so one step is:
x_new = x - η·2x = (1 - 2η)·xEach step multiplies x by the same factor, 1 - 2η:
- η = 0.1 → factor 0.8: shrinks steadily
- η = 0.5 → factor 0: lands exactly on the minimum in one step
- η = 0.9 → factor -0.8: flips sign each step but shrinks (oscillating)
- η = 1.1 → factor -1.2: flips and grows: divergence
for rate in [0.01, 0.1, 0.5, 0.9, 1.1]:
x = 5.0
history = []
for step in range(6):
x = x - rate * 2 * x
history.append(round(x, 2))
print(f"η = {rate:<4}: {history}")Narrow valleys: zigzags
A real loss curves differently in different directions. In a narrow valley, the steep direction limits the learning rate, but the gentle direction then crawls. Steps bounce across the valley while creeping along it.
Try the valley f = 0.5x² + 5y² below. With η around 0.17 the path zigzags. Above 0.2 it diverges in the steep y direction. Below 0.05 it barely moves along x:
Tip: click anywhere on the map to start from there.
This is one of the main motivations for momentum and Adam, which give the gentle direction more speed and damp the bouncing.
Learning rate schedules
A common trick is to start with a larger learning rate to make fast progress, then shrink it to settle precisely into the minimum.
def f_prime(x):
return 2 * (x - 3)
x = 0.0
rate = 0.9 # aggressive at first
for step in range(1, 21):
x = x - rate * f_prime(x)
rate = rate * 0.8 # exponential decay: 20% smaller each step
if step % 4 == 0:
print(f"step {step:>2}: x = {x:.5f}, learning rate now {rate:.4f}")Many large models also use a warm-up: the learning rate starts tiny for the first few thousand steps and rises to its target, because the gradients of a freshly initialised network are unreliable.
Common mistakes
Compare five learning rates and identify slow, ideal, oscillating and diverging behaviour.
# Gradient descent on f(x) = x**2, whose derivative is 2 * x. # For each learning rate, start at x = 5.0, run 9 steps of # x = x - rate * 2 * x # and print the final x rounded to 4 places. rates = [0.01, 0.1, 0.5, 0.9, 1.1] # Output must be exactly: # rate 0.01: x = 4.1687 # rate 0.1: x = 0.6711 # rate 0.5: x = 0.0 # rate 0.9: x = -0.6711 # rate 1.1: x = -25.7989
During training, your loss jumps up and down wildly and then becomes nan. What is the most likely fix?
Summary
- The learning rate is a hyperparameter that sets the step size.
- Too small: slow. Just right: fast. Too big: oscillation or divergence.
- For a bowl with curvature a, steps converge only if
η < 1/a: steep losses need small steps. - Narrow valleys cause zigzags, which momentum and Adam reduce.
- Schedules (decay, warm-up) change η during training. Search η by factors of 10.
Next: what can go wrong in bumpy landscapes, local minima and saddle points.
Finished reading? Tick it off to track your progress.