Velamuvi AIAcademy

Lesson 14 of 16 · Learning by Descent

Learning Rate

See how the learning rate controls gradient descent: slow progress when it is too small, zigzags and divergence when it is too big, the maths of when it converges, narrow valleys, learning rate schedules and how to pick a good value.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What happens when the learning rate is too small, too big or just right
  • Why too big a step can make the loss explode
  • How the curvature of the loss limits the learning rate
  • Why narrow valleys cause zigzagging
  • How learning rate schedules and simple searches find a good value

The most important setting

The learning rate η decides how far each gradient descent step goes. It is a hyperparameter: something you choose, not something the model learns.

Three behaviours

Try it on f(x) = x², starting at x = 4. Slide the learning rate from small to large:

-6-4-20246051015202530
f(x) = x²
start: x = 4after 10 steps: x = 0.429, f(x) = 0.184
Small η crawls. Around 0.3 to 0.5 it converges fast. Above 0.5 it jumps back and forth across the valley. Above 1 every jump lands higher than the last, and it diverges.
Learning rateWhat happens
too smallevery step is safe but tiny, so training takes forever
just rightfast, steady progress to the minimum
a bit too bigovershoots, bouncing from side to side but still settling
far too bigeach step lands higher than the last: the loss diverges to infinity

Why too big explodes

For f(x) = x² the slope is 2x, so one step is:

x_new = x - η·2x = (1 - 2η)·x

Each step multiplies x by the same factor, 1 - 2η:

  • η = 0.1 → factor 0.8: shrinks steadily
  • η = 0.5 → factor 0: lands exactly on the minimum in one step
  • η = 0.9 → factor -0.8: flips sign each step but shrinks (oscillating)
  • η = 1.1 → factor -1.2: flips and grows: divergence
Try it yourselfPython
for rate in [0.01, 0.1, 0.5, 0.9, 1.1]:
    x = 5.0
    history = []
    for step in range(6):
        x = x - rate * 2 * x
        history.append(round(x, 2))
    print(f"η = {rate:<4}: {history}")

Narrow valleys: zigzags

A real loss curves differently in different directions. In a narrow valley, the steep direction limits the learning rate, but the gentle direction then crawls. Steps bounce across the valley while creeping along it.

Try the valley f = 0.5x² + 5y² below. With η around 0.17 the path zigzags. Above 0.2 it diverges in the steep y direction. Below 0.05 it barely moves along x:

start (-2.8, 1.2): f = 11.12after 25 steps: (-0.027, 0), f = 0
No single learning rate is good for both directions: steep across the valley, gentle along it.

Tip: click anywhere on the map to start from there.

This is one of the main motivations for momentum and Adam, which give the gentle direction more speed and damp the bouncing.

Learning rate schedules

A common trick is to start with a larger learning rate to make fast progress, then shrink it to settle precisely into the minimum.

Try it yourselfPython
def f_prime(x):
    return 2 * (x - 3)

x = 0.0
rate = 0.9                       # aggressive at first
for step in range(1, 21):
    x = x - rate * f_prime(x)
    rate = rate * 0.8            # exponential decay: 20% smaller each step
    if step % 4 == 0:
        print(f"step {step:>2}: x = {x:.5f}, learning rate now {rate:.4f}")

Many large models also use a warm-up: the learning rate starts tiny for the first few thousand steps and rises to its target, because the gradients of a freshly initialised network are unreliable.

Common mistakes

Practice exercise

Compare five learning rates and identify slow, ideal, oscillating and diverging behaviour.

Your solutionPython
# Gradient descent on f(x) = x**2, whose derivative is 2 * x.
# For each learning rate, start at x = 5.0, run 9 steps of
#   x = x - rate * 2 * x
# and print the final x rounded to 4 places.

rates = [0.01, 0.1, 0.5, 0.9, 1.1]

# Output must be exactly:
# rate 0.01: x = 4.1687
# rate 0.1: x = 0.6711
# rate 0.5: x = 0.0
# rate 0.9: x = -0.6711
# rate 1.1: x = -25.7989
Knowledge check

During training, your loss jumps up and down wildly and then becomes nan. What is the most likely fix?

Summary

  • The learning rate is a hyperparameter that sets the step size.
  • Too small: slow. Just right: fast. Too big: oscillation or divergence.
  • For a bowl with curvature a, steps converge only if η < 1/a: steep losses need small steps.
  • Narrow valleys cause zigzags, which momentum and Adam reduce.
  • Schedules (decay, warm-up) change η during training. Search η by factors of 10.

Next: what can go wrong in bumpy landscapes, local minima and saddle points.

Finished reading? Tick it off to track your progress.