Velamuvi AIAcademy

Lesson 15 of 16 · Learning by Descent

Local Minima and Saddle Points

Learn what can stop gradient descent: local versus global minima, saddle points, plateaus, convex and non-convex losses, and why deep networks still train well, with interactive landscapes.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • The difference between a local and a global minimum
  • What a saddle point is and why gradient descent slows down near one
  • What convex functions are and why they are easy to minimise
  • Why the starting point decides where gradient descent ends
  • Why deep networks train well despite non-convex losses

Flat is not always finished

Gradient descent stops moving when the gradient is zero. But a zero gradient only says "flat here". There are several kinds of flat point:

Where you start decides where you end

The curve below has a deep valley on the left (the global minimum, near x = -2.6) and a shallow one on the right (a local minimum, near x = 2.3). Move the start point across the hilltop near x = 0.3 and watch gradient descent switch valleys:

-4-3-2-101234-2-10123456
f(x)
start: x = 3after 30 steps: x = 2.262, f(x) = 1.144
Starting right of the hilltop, descent gets stuck in the shallow valley (f ≈ 1.14). Starting left of it, descent finds the deep valley (f ≈ -0.81).
Try it yourselfPython
def f(x):
    return 0.05 * x**4 - 0.6 * x**2 + 0.4 * x + 2

def f_prime(x):
    return 0.2 * x**3 - 1.2 * x + 0.4

for start in [-3.0, -1.0, 0.5, 1.0, 3.0]:
    x = start
    for step in range(500):
        x = x - 0.1 * f_prime(x)
    print(f"start {start:>4}: ends at x = {x:.3f}, f = {f(x):.3f}")

Gradient descent only sees the local slope. It has no idea a deeper valley exists elsewhere.

Convex functions: no traps

Some functions have no traps at all.

The squared-error loss of linear regression is convex, and so is the loss of logistic regression. That is why those models train reliably from any starting point. The loss of a neural network is non-convex: full of hills, valleys and saddles.

Saddle points

In two or more dimensions there is a third kind of flat point.

On the saddle f = x² - y², the ground rises along x and falls along y. Start exactly on the x axis (y = 0) and gradient descent slides into the saddle and stops, because the gradient there is zero. Nudge the start slightly off the axis and it eventually escapes downhill:

start (2.8, 0): f = 7.84after 30 steps: (0.003, 0), f = 0
Start at y = 0 and descent gets stuck at the saddle point (0, 0). Set start y to 0.1 and it slowly peels away towards lower ground.

Tip: click anywhere on the map to start from there.

Try it yourselfPython
for start_y in [0.0, 0.01]:
    x, y = 2.0, start_y
    for step in range(30):
        grad_x, grad_y = 2 * x, -2 * y        # gradient of x² - y²
        x = x - 0.1 * grad_x
        y = y - 0.1 * grad_y
    print(f"start (2, {start_y}): after 30 steps ({x:.4f}, {y:.4f}), f = {x*x - y*y:.4f}")

Near a saddle the gradient is tiny, so progress slows to a crawl even when escape is possible. A large flat region with tiny gradients is called a plateau, and it has the same effect.

Why deep networks still train

Neural network losses are non-convex, so why does training work so well? Research suggests:

  • Saddles, not bad minima, are the main obstacle. In millions of dimensions, a flat point is a minimum only if the loss curves upwards in every direction, which is rare. Most flat points are saddles, which noisy updates eventually escape.
  • Most local minima are good enough. In large networks, the local minima that training finds tend to have losses close to the global minimum.
  • Noise helps. Mini-batch gradients are slightly random, which shakes the parameters out of saddles and shallow dips.

Common mistakes

Practice exercise

Run gradient descent from two starting points and see it land in two different minima. Which one is the global minimum?

Your solutionPython
def f(x):
    return x**4 - 4 * x**2 + x

def f_prime(x):
    return 4 * x**3 - 8 * x + 1

# Run 200 steps of gradient descent with rate 0.01 from start = -2.0
# and from start = 2.0. For each, print the final x and f(x) rounded to 3 places.

# Output must be exactly:
# start -2.0: x = -1.473, f(x) = -5.444
# start 2.0: x = 1.347, f(x) = -2.619
Knowledge check

Why does gradient descent reliably find the best answer for linear regression with squared error?

Summary

  • A critical point has zero gradient: minimum, maximum or saddle point.
  • A local minimum can trap gradient descent above the global minimum, and the start point decides which one you reach.
  • Convex losses have one bowl and no traps. Neural network losses are non-convex.
  • Near saddles and plateaus, tiny gradients slow training.
  • Noise, momentum, good initialisation and schedules help deep networks train well anyway.

Next: a short look at the other half of calculus, integrals.

Finished reading? Tick it off to track your progress.