Lesson 6 of 16 · Many Variables
The Gradient
Understand the gradient: the vector of all partial derivatives. See why it points uphill, why its length is the steepness, why it is zero at minimums, and how the gradient of a loss trains a model, with an interactive contour map.
- 7 min read
- Intermediate
- Reviewed
- What the gradient of a function is
- Why the gradient points in the direction of steepest increase
- What the length of the gradient tells you
- How to read a contour map and its gradient arrows
- How to compute the gradient of a model's loss with NumPy
All the partial derivatives together
In the last lesson, f(x, y) had two partial derivatives, one per input. Put them side by side in a vector and you get the gradient.
f(x, y) = x² + 4y²
∂f/∂x = 2x, ∂f/∂y = 8y
∇f(x, y) = (2x, 8y) at (1, 1): ∇f = (2, 8)A function with a million inputs has a gradient with a million entries, one for each input.
Reading a contour map
A contour map shows a landscape from above, like a hiking map. Each line joins points at the same height.
Below is f(x, y) = x² + 4y², a bowl with its lowest point at (0, 0). The small pink arrows show the gradient. Move the yellow point with the sliders: the long pink arrow is the gradient there and the dashed green arrow points the opposite way, downhill.
Three facts to notice:
- The gradient points uphill, in the steepest direction. Its negative, -∇f, points steepest downhill.
- Its length is the steepness. Near the centre the ground is almost flat and the gradient is short.
- It is zero at the bottom, where the ground is flat in every direction.
Computing gradients in Python
import numpy as np
def gradient(x, y):
return np.array([2 * x, 8 * y]) # (∂f/∂x, ∂f/∂y) for f = x² + 4y²
for point in [(1, 1), (2, 0), (0, -1), (0, 0)]:
g = gradient(point[0], point[1])
steepness = np.linalg.norm(g) # the magnitude (length)
print(f"at {point}: gradient {g}, steepness {steepness:.3f}")At (2, 0) the gradient is (4, 0): uphill is straight along x. At (0, 0) it is (0, 0): the bottom.
The gradient of a loss
For a model, the landscape is the loss and the position is the set of parameter values. The gradient of the loss says how to change every parameter at once to make the loss grow fastest, so we move the opposite way.
import numpy as np
x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([3.0, 5.0, 7.0, 9.0])
def loss_and_gradient(w, b):
error = w * x + b - y
loss = np.mean(error ** 2)
grad = np.array([np.mean(2 * error * x), # ∂L/∂w
np.mean(2 * error)]) # ∂L/∂b
return loss, grad
for w, b in [(1.0, 0.0), (1.5, 0.5), (2.0, 1.0)]:
loss, grad = loss_and_gradient(w, b)
print(f"w={w}, b={b}: loss {loss:.3f}, gradient {np.round(grad, 3) + 0.0}")As the parameters approach the best values (w = 2, b = 1), the loss falls and the gradient shrinks to (0, 0). A zero gradient is the signal that training has reached a flat point.
Common mistakes
Compute the gradient of a model's loss. Both partial derivatives are negative, so the model should increase w and b.
import numpy as np x = np.array([1.0, 2.0, 3.0]) y = np.array([2.0, 4.0, 6.0]) w, b = 1.0, 0.0 # The model predicts w * x + b. Its loss is the mean of (prediction - y)**2. # Print the loss and both partial derivatives, each rounded to 3 places: # dL/dw = mean of 2 * error * x # dL/db = mean of 2 * error # where error = prediction - y. # Output must be exactly: # loss: 4.667 # dL/dw: -9.333 # dL/db: -4.0
The gradient of a loss at the current weights is (3, -2). Which way should the weights move to reduce the loss fastest?
Summary
- The gradient ∇f collects all the partial derivatives into one vector.
- It points in the direction of steepest increase, and its length is the steepness.
- It crosses contour lines at right angles and is zero at flat points such as minimums.
- The negative gradient points steepest downhill, the direction training moves.
- For a model, the gradient of the loss tells every parameter how to change at once.
Next: the rate of change in any direction you choose.
Finished reading? Tick it off to track your progress.