Lesson 7 of 16 · Many Variables
Directional Change
Learn how fast a function changes in any direction: unit vectors, the directional derivative as a dot product with the gradient, why the gradient is the steepest direction, and the linear approximation that makes gradient descent work.
- 8 min read
- Intermediate
- Reviewed
- What a unit vector is
- How to compute the rate of change in any direction
- Why the gradient direction is the steepest and the contour direction is flat
- How the gradient gives a linear approximation of a function
- Why steepest downhill does not always point at the minimum
Change in any direction
Partial derivatives measure change along the x axis and along the y axis. But you can walk in any direction: north-east, slightly west of south, anywhere. How fast does the function change then?
First you need a way to describe a direction.
Any vector can be turned into a unit vector by dividing it by its length: (3, 4) has length √(9 + 16) = 5, so its direction is (0.6, 0.8).
The directional derivative
rate of change in direction u = ∇f · u = (∂f/∂x)·u₁ + (∂f/∂y)·u₂It mixes the two partial derivatives in proportion to how much you move along x and along y. Moving east, u = (1, 0), gives back exactly ∂f/∂x, as it should.
import math
import numpy as np
gradient = np.array([3.0, 4.0]) # the gradient at some point
for name, angle in [("east", 0), ("north-east", 45), ("north", 90), ("west", 180), ("south", 270)]:
u = np.array([math.cos(math.radians(angle)), math.sin(math.radians(angle))])
rate = gradient @ u # @ is the dot product
print(f"{name:<11} rate of change = {round(rate, 3) + 0.0}")Steepest, flattest and in between
Try every direction and three special ones stand out:
| Direction | Rate of change | Why |
|---|---|---|
| along the gradient | largest: the length of ∇f | the dot product is biggest when the vectors point the same way |
| opposite the gradient | most negative: minus the length | steepest downhill |
| at right angles to the gradient | zero | you are walking along a contour line |
For the gradient (3, 4), the steepest rate is 5 (its length), in the direction (0.6, 0.8). The steepest descent is -5, in direction (-0.6, -0.8). And along (0.8, -0.6) the function does not change at all.
Steepest downhill is not always "towards the bottom"
In a round bowl, the downhill arrows point straight at the lowest point. In a long, narrow valley they do not: they point mostly across the valley, towards the nearest steep wall.
This is why gradient descent can zigzag across narrow valleys instead of heading straight for the bottom, a problem you will see in the learning rate lesson.
The linear approximation
Here is the most useful consequence. For a small step Δ = (Δx, Δy) from a point p:
f(p + Δ) ≈ f(p) + ∇f · ΔClose to any point, a smooth function behaves almost like a flat, tilted plane, and the gradient describes the tilt.
import numpy as np
def f(x, y):
return x**2 + 4 * y**2
p = np.array([1.0, 1.0])
grad = np.array([2.0, 8.0]) # ∇f at (1, 1)
for step in [(0.01, 0.01), (0.1, -0.05), (-0.5, 0.5)]:
s = np.array(step)
actual = f(p[0] + s[0], p[1] + s[1]) - f(p[0], p[1])
predicted = grad @ s
print(f"step {step}: actual change {actual:.4f}, linear estimate {predicted:.4f}")For the tiny step the estimate is nearly perfect. For the big step it is far off, because the bowl curves away from the plane.
Common mistakes
Compute directional derivatives and confirm that the steepest rate equals the length of the gradient.
import numpy as np
gradient = np.array([3.0, 4.0])
directions = {
"east": np.array([1.0, 0.0]),
"north": np.array([0.0, 1.0]),
"up the gradient": np.array([0.6, 0.8]),
"down the gradient": np.array([-0.6, -0.8]),
}
# For each direction, print the rate of change (gradient · direction),
# rounded to 3 places. Then print the steepness: the length of the gradient.
# Output must be exactly:
# east: 3.0
# north: 4.0
# up the gradient: 5.0
# down the gradient: -5.0
# steepness: 5.0
The gradient at a point is (5, 0). In which direction does the function not change at all?
Summary
- A unit vector describes a direction. Divide any vector by its length to get one.
- The directional derivative in direction u is ∇f · u.
- It is largest along the gradient (equal to its length), most negative opposite it, and zero along contour lines.
- The linear approximation f(p + Δ) ≈ f(p) + ∇f · Δ is accurate for small steps.
- A small step against the gradient always lowers the function: the reason gradient descent works.
Next: what to do when a function has many outputs, or when you need second derivatives.
Finished reading? Tick it off to track your progress.