Velamuvi AIAcademy

Lesson 6 of 16 · Many Variables

The Gradient

Understand the gradient: the vector of all partial derivatives. See why it points uphill, why its length is the steepness, why it is zero at minimums, and how the gradient of a loss trains a model, with an interactive contour map.

  • 7 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What the gradient of a function is
  • Why the gradient points in the direction of steepest increase
  • What the length of the gradient tells you
  • How to read a contour map and its gradient arrows
  • How to compute the gradient of a model's loss with NumPy

All the partial derivatives together

In the last lesson, f(x, y) had two partial derivatives, one per input. Put them side by side in a vector and you get the gradient.

f(x, y) = x² + 4y²

∂f/∂x = 2x,   ∂f/∂y = 8y

∇f(x, y) = (2x, 8y)          at (1, 1): ∇f = (2, 8)

A function with a million inputs has a gradient with a million entries, one for each input.

Reading a contour map

A contour map shows a landscape from above, like a hiking map. Each line joins points at the same height.

Below is f(x, y) = x² + 4y², a bowl with its lowest point at (0, 0). The small pink arrows show the gradient. Move the yellow point with the sliders: the long pink arrow is the gradient there and the dashed green arrow points the opposite way, downhill.

at (1, 1): f = 5gradient = (2, 8)steepness = 8.246
The gradient always crosses the contour lines at right angles and points uphill. It is longest where the lines are close together and zero at the bottom of the bowl.

Three facts to notice:

  1. The gradient points uphill, in the steepest direction. Its negative, -∇f, points steepest downhill.
  2. Its length is the steepness. Near the centre the ground is almost flat and the gradient is short.
  3. It is zero at the bottom, where the ground is flat in every direction.

Computing gradients in Python

Try it yourselfPython
import numpy as np

def gradient(x, y):
    return np.array([2 * x, 8 * y])      # (∂f/∂x, ∂f/∂y) for f = x² + 4y²

for point in [(1, 1), (2, 0), (0, -1), (0, 0)]:
    g = gradient(point[0], point[1])
    steepness = np.linalg.norm(g)          # the magnitude (length)
    print(f"at {point}: gradient {g}, steepness {steepness:.3f}")

At (2, 0) the gradient is (4, 0): uphill is straight along x. At (0, 0) it is (0, 0): the bottom.

The gradient of a loss

For a model, the landscape is the loss and the position is the set of parameter values. The gradient of the loss says how to change every parameter at once to make the loss grow fastest, so we move the opposite way.

Try it yourselfPython
import numpy as np

x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([3.0, 5.0, 7.0, 9.0])

def loss_and_gradient(w, b):
    error = w * x + b - y
    loss = np.mean(error ** 2)
    grad = np.array([np.mean(2 * error * x),    # ∂L/∂w
                     np.mean(2 * error)])       # ∂L/∂b
    return loss, grad

for w, b in [(1.0, 0.0), (1.5, 0.5), (2.0, 1.0)]:
    loss, grad = loss_and_gradient(w, b)
    print(f"w={w}, b={b}: loss {loss:.3f}, gradient {np.round(grad, 3) + 0.0}")

As the parameters approach the best values (w = 2, b = 1), the loss falls and the gradient shrinks to (0, 0). A zero gradient is the signal that training has reached a flat point.

Common mistakes

Practice exercise

Compute the gradient of a model's loss. Both partial derivatives are negative, so the model should increase w and b.

Your solutionPython
import numpy as np

x = np.array([1.0, 2.0, 3.0])
y = np.array([2.0, 4.0, 6.0])
w, b = 1.0, 0.0

# The model predicts w * x + b. Its loss is the mean of (prediction - y)**2.
# Print the loss and both partial derivatives, each rounded to 3 places:
#   dL/dw = mean of 2 * error * x
#   dL/db = mean of 2 * error
# where error = prediction - y.

# Output must be exactly:
# loss: 4.667
# dL/dw: -9.333
# dL/db: -4.0
Knowledge check

The gradient of a loss at the current weights is (3, -2). Which way should the weights move to reduce the loss fastest?

Summary

  • The gradient ∇f collects all the partial derivatives into one vector.
  • It points in the direction of steepest increase, and its length is the steepness.
  • It crosses contour lines at right angles and is zero at flat points such as minimums.
  • The negative gradient points steepest downhill, the direction training moves.
  • For a model, the gradient of the loss tells every parameter how to change at once.

Next: the rate of change in any direction you choose.

Finished reading? Tick it off to track your progress.