Velamuvi AIAcademy

Lesson 13 of 16 · Learning by Descent

Gradient Descent

Understand gradient descent, the algorithm that trains almost every machine learning model: the update rule, interactive pictures in one and two dimensions, and a line fitted to data from scratch with NumPy.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • The idea of gradient descent, as walking downhill in fog
  • The update rule: new = old - learning rate × gradient
  • How gradient descent behaves in one and two dimensions
  • How to train a linear model with gradient descent from scratch
  • When to stop, and how batches change the picture

Walking downhill in fog

You are on a hillside in thick fog and want to reach the valley floor. You cannot see the valley, but you can feel the slope under your feet. A sensible plan:

  1. Feel which way is steepest downhill.
  2. Take a small step that way.
  3. Repeat until the ground is flat.

That is gradient descent. The hillside is the loss, your position is the model's parameters, and "feeling the slope" is computing the gradient.

The update rule

Each step updates every parameter at once:

new parameters = old parameters - η × gradient of the loss

In one dimension, with a single parameter x, the rule is x ← x - η·f'(x). If the slope is positive, x decreases. If it is negative, x increases. Either way, it moves downhill, and steep slopes give bigger steps.

-20246805101520
f(x) = (x - 3)² + 1
start: x = -1after 5 steps: x = 2.689, f(x) = 1.097
The yellow dot is the start and the pink dot is where you are after the chosen number of steps. Steps are big where the curve is steep and shrink as it flattens near the minimum at x = 3.
Try it yourselfPython
def f(x):
    return (x - 3) ** 2 + 1

def f_prime(x):
    return 2 * (x - 3)

x = -1.0              # start
rate = 0.2            # learning rate

for step in range(1, 11):
    x = x - rate * f_prime(x)
    print(f"step {step:>2}: x = {x:.4f}, f(x) = {f(x):.4f}")

Every step closes 40% of the remaining gap to 3, so the steps get smaller automatically as the slope flattens.

In two dimensions

With two parameters, the gradient is a vector and the step goes against it. Click anywhere on the map to start there, and change the learning rate and the number of steps:

start (-2.5, 2): f = 14.25after 15 steps: (-0.088, 0.001), f = 0.008
Gradient descent on the bowl f = x² + 2y². Each step crosses the contour lines at right angles, heading for the minimum at (0, 0).

Tip: click anywhere on the map to start from there.

Training a model from scratch

Now the real thing: fit a line prediction = w·x + b to data by minimising the mean squared error. You computed this gradient in the gradient lesson. Gradient descent just uses it again and again:

Try it yourselfPython
import numpy as np

x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.1, 4.9, 7.2, 8.8, 11.1])     # roughly y = 2x + 1, with noise

w, b = 0.0, 0.0                               # start from a bad guess
rate = 0.02

for step in range(1001):
    error = w * x + b - y
    loss = np.mean(error ** 2)
    grad_w = np.mean(2 * error * x)            # ∂L/∂w
    grad_b = np.mean(2 * error)                # ∂L/∂b

    if step in [0, 10, 50, 200, 1000]:
        print(f"step {step:>4}: loss {loss:7.4f}, w = {w:.3f}, b = {b:.3f}")

    w = w - rate * grad_w                      # the update rule
    b = b - rate * grad_b

The loss falls from 57 to about 0.02, and the parameters settle near w = 1.99 and b = 1.05: the best line through the noisy data. That loop, made bigger, is how every neural network is trained.

When to stop

Gradient descent never lands exactly on the minimum. Common stopping rules:

  • after a fixed number of steps (often called epochs when each step uses the whole dataset)
  • when the gradient becomes very small (the ground is flat)
  • when the loss stops improving, or starts getting worse on data held back for checking (early stopping)

Common mistakes

Practice exercise

Run gradient descent on a simple curve and watch it converge to the minimum at x = 3.

Your solutionPython
def f(x):
    return (x - 3) ** 2 + 1

def f_prime(x):
    return 2 * (x - 3)

x = 0.0
rate = 0.1

# Run 50 steps of gradient descent: x = x - rate * f_prime(x).
# After every 10th step, print x and f(x), each rounded to 4 places.

# Output must be exactly:
# step 10: x = 2.6779, f(x) = 1.1038
# step 20: x = 2.9654, f(x) = 1.0012
# step 30: x = 2.9963, f(x) = 1.0
# step 40: x = 2.9996, f(x) = 1.0
# step 50: x = 3.0, f(x) = 1.0
Knowledge check

A parameter has value 2.0, the gradient of the loss with respect to it is 4.0, and the learning rate is 0.1. What is its value after one gradient descent step?

Summary

  • Gradient descent repeatedly steps against the gradient: θ ← θ - η·∇L.
  • Steps are big where the loss is steep and shrink near a flat minimum.
  • It works the same in 1, 2 or a billion dimensions.
  • Training a model means running this loop on the gradient of its loss.
  • Real training uses mini-batches (SGD) and smarter variants such as momentum and Adam.

Next: the learning rate, the most important setting in gradient descent.

Finished reading? Tick it off to track your progress.