Lesson 13 of 16 · Learning by Descent
Gradient Descent
Understand gradient descent, the algorithm that trains almost every machine learning model: the update rule, interactive pictures in one and two dimensions, and a line fitted to data from scratch with NumPy.
- 8 min read
- Intermediate
- Reviewed
- The idea of gradient descent, as walking downhill in fog
- The update rule: new = old - learning rate × gradient
- How gradient descent behaves in one and two dimensions
- How to train a linear model with gradient descent from scratch
- When to stop, and how batches change the picture
Walking downhill in fog
You are on a hillside in thick fog and want to reach the valley floor. You cannot see the valley, but you can feel the slope under your feet. A sensible plan:
- Feel which way is steepest downhill.
- Take a small step that way.
- Repeat until the ground is flat.
That is gradient descent. The hillside is the loss, your position is the model's parameters, and "feeling the slope" is computing the gradient.
The update rule
Each step updates every parameter at once:
new parameters = old parameters - η × gradient of the lossIn one dimension, with a single parameter x, the rule is x ← x - η·f'(x). If the slope is positive, x decreases. If it is negative, x increases. Either way, it moves downhill, and steep slopes give bigger steps.
def f(x):
return (x - 3) ** 2 + 1
def f_prime(x):
return 2 * (x - 3)
x = -1.0 # start
rate = 0.2 # learning rate
for step in range(1, 11):
x = x - rate * f_prime(x)
print(f"step {step:>2}: x = {x:.4f}, f(x) = {f(x):.4f}")Every step closes 40% of the remaining gap to 3, so the steps get smaller automatically as the slope flattens.
In two dimensions
With two parameters, the gradient is a vector and the step goes against it. Click anywhere on the map to start there, and change the learning rate and the number of steps:
Tip: click anywhere on the map to start from there.
Training a model from scratch
Now the real thing: fit a line prediction = w·x + b to data by minimising the mean squared error. You computed this gradient in the gradient lesson. Gradient descent just uses it again and again:
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([3.1, 4.9, 7.2, 8.8, 11.1]) # roughly y = 2x + 1, with noise
w, b = 0.0, 0.0 # start from a bad guess
rate = 0.02
for step in range(1001):
error = w * x + b - y
loss = np.mean(error ** 2)
grad_w = np.mean(2 * error * x) # ∂L/∂w
grad_b = np.mean(2 * error) # ∂L/∂b
if step in [0, 10, 50, 200, 1000]:
print(f"step {step:>4}: loss {loss:7.4f}, w = {w:.3f}, b = {b:.3f}")
w = w - rate * grad_w # the update rule
b = b - rate * grad_bThe loss falls from 57 to about 0.02, and the parameters settle near w = 1.99 and b = 1.05: the best line through the noisy data. That loop, made bigger, is how every neural network is trained.
When to stop
Gradient descent never lands exactly on the minimum. Common stopping rules:
- after a fixed number of steps (often called epochs when each step uses the whole dataset)
- when the gradient becomes very small (the ground is flat)
- when the loss stops improving, or starts getting worse on data held back for checking (early stopping)
Common mistakes
Run gradient descent on a simple curve and watch it converge to the minimum at x = 3.
def f(x):
return (x - 3) ** 2 + 1
def f_prime(x):
return 2 * (x - 3)
x = 0.0
rate = 0.1
# Run 50 steps of gradient descent: x = x - rate * f_prime(x).
# After every 10th step, print x and f(x), each rounded to 4 places.
# Output must be exactly:
# step 10: x = 2.6779, f(x) = 1.1038
# step 20: x = 2.9654, f(x) = 1.0012
# step 30: x = 2.9963, f(x) = 1.0
# step 40: x = 2.9996, f(x) = 1.0
# step 50: x = 3.0, f(x) = 1.0
A parameter has value 2.0, the gradient of the loss with respect to it is 4.0, and the learning rate is 0.1. What is its value after one gradient descent step?
Summary
- Gradient descent repeatedly steps against the gradient:
θ ← θ - η·∇L. - Steps are big where the loss is steep and shrink near a flat minimum.
- It works the same in 1, 2 or a billion dimensions.
- Training a model means running this loop on the gradient of its loss.
- Real training uses mini-batches (SGD) and smarter variants such as momentum and Adam.
Next: the learning rate, the most important setting in gradient descent.
Finished reading? Tick it off to track your progress.