Velamuvi AIAcademy

Lesson 7 of 16 · Many Variables

Directional Change

Learn how fast a function changes in any direction: unit vectors, the directional derivative as a dot product with the gradient, why the gradient is the steepest direction, and the linear approximation that makes gradient descent work.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What a unit vector is
  • How to compute the rate of change in any direction
  • Why the gradient direction is the steepest and the contour direction is flat
  • How the gradient gives a linear approximation of a function
  • Why steepest downhill does not always point at the minimum

Change in any direction

Partial derivatives measure change along the x axis and along the y axis. But you can walk in any direction: north-east, slightly west of south, anywhere. How fast does the function change then?

First you need a way to describe a direction.

Any vector can be turned into a unit vector by dividing it by its length: (3, 4) has length √(9 + 16) = 5, so its direction is (0.6, 0.8).

The directional derivative

rate of change in direction u  =  ∇f · u  =  (∂f/∂x)·u₁ + (∂f/∂y)·u₂

It mixes the two partial derivatives in proportion to how much you move along x and along y. Moving east, u = (1, 0), gives back exactly ∂f/∂x, as it should.

Try it yourselfPython
import math
import numpy as np

gradient = np.array([3.0, 4.0])          # the gradient at some point

for name, angle in [("east", 0), ("north-east", 45), ("north", 90), ("west", 180), ("south", 270)]:
    u = np.array([math.cos(math.radians(angle)), math.sin(math.radians(angle))])
    rate = gradient @ u                    # @ is the dot product
    print(f"{name:<11} rate of change = {round(rate, 3) + 0.0}")

Steepest, flattest and in between

Try every direction and three special ones stand out:

DirectionRate of changeWhy
along the gradientlargest: the length of ∇fthe dot product is biggest when the vectors point the same way
opposite the gradientmost negative: minus the lengthsteepest downhill
at right angles to the gradientzeroyou are walking along a contour line

For the gradient (3, 4), the steepest rate is 5 (its length), in the direction (0.6, 0.8). The steepest descent is -5, in direction (-0.6, -0.8). And along (0.8, -0.6) the function does not change at all.

Steepest downhill is not always "towards the bottom"

In a round bowl, the downhill arrows point straight at the lowest point. In a long, narrow valley they do not: they point mostly across the valley, towards the nearest steep wall.

A narrow valley, f = 0.5x² + 5y². The green downhill arrows point mostly across the valley (the y direction), not along it towards the minimum at (0, 0).

This is why gradient descent can zigzag across narrow valleys instead of heading straight for the bottom, a problem you will see in the learning rate lesson.

The linear approximation

Here is the most useful consequence. For a small step Δ = (Δx, Δy) from a point p:

f(p + Δ)  ≈  f(p) + ∇f · Δ

Close to any point, a smooth function behaves almost like a flat, tilted plane, and the gradient describes the tilt.

Try it yourselfPython
import numpy as np

def f(x, y):
    return x**2 + 4 * y**2

p = np.array([1.0, 1.0])
grad = np.array([2.0, 8.0])                 # ∇f at (1, 1)

for step in [(0.01, 0.01), (0.1, -0.05), (-0.5, 0.5)]:
    s = np.array(step)
    actual = f(p[0] + s[0], p[1] + s[1]) - f(p[0], p[1])
    predicted = grad @ s
    print(f"step {step}: actual change {actual:.4f}, linear estimate {predicted:.4f}")

For the tiny step the estimate is nearly perfect. For the big step it is far off, because the bowl curves away from the plane.

Common mistakes

Practice exercise

Compute directional derivatives and confirm that the steepest rate equals the length of the gradient.

Your solutionPython
import numpy as np

gradient = np.array([3.0, 4.0])

directions = {
    "east": np.array([1.0, 0.0]),
    "north": np.array([0.0, 1.0]),
    "up the gradient": np.array([0.6, 0.8]),
    "down the gradient": np.array([-0.6, -0.8]),
}

# For each direction, print the rate of change (gradient · direction),
# rounded to 3 places. Then print the steepness: the length of the gradient.

# Output must be exactly:
# east: 3.0
# north: 4.0
# up the gradient: 5.0
# down the gradient: -5.0
# steepness: 5.0
Knowledge check

The gradient at a point is (5, 0). In which direction does the function not change at all?

Summary

  • A unit vector describes a direction. Divide any vector by its length to get one.
  • The directional derivative in direction u is ∇f · u.
  • It is largest along the gradient (equal to its length), most negative opposite it, and zero along contour lines.
  • The linear approximation f(p + Δ) ≈ f(p) + ∇f · Δ is accurate for small steps.
  • A small step against the gradient always lowers the function: the reason gradient descent works.

Next: what to do when a function has many outputs, or when you need second derivatives.

Finished reading? Tick it off to track your progress.