Lesson 8 of 16 · Many Variables
The Jacobian and Hessian
Understand second derivatives and curvature, the second derivative test, the Hessian matrix of a function of many variables, and the Jacobian matrix of a function with many outputs, with NumPy and SymPy examples.
- 10 min read
- Intermediate
- Reviewed
- What a second derivative measures: curvature
- How to tell a minimum from a maximum with the second derivative test
- What the Hessian matrix is and what it says about a landscape
- What the Jacobian matrix is, for functions with several outputs
- Where Jacobians and Hessians appear in machine learning
The derivative of the derivative
The derivative f'(x) is itself a function, so you can differentiate it again. The result is the second derivative, f''(x).
For f(x) = x³ - 3x:
f'(x) = 3x² - 3
f''(x) = 6x| f''(x) | The curve... | Shape |
|---|---|---|
| positive | bends upwards, slope increasing | like a bowl ∪ |
| negative | bends downwards, slope decreasing | like a hill ∩ |
| zero | is not bending at that point | possibly changing from one to the other |
The second derivative test
A flat point, where f'(x) = 0, is called a critical point. The second derivative tells you what kind it is:
f'(x) = 0andf''(x) > 0→ bottom of a bowl → minimumf'(x) = 0andf''(x) < 0→ top of a hill → maximumf'(x) = 0andf''(x) = 0→ the test cannot tell
A numerical second derivative uses three points:
def f(x):
return x**3 - 3 * x
def second_derivative(f, x, h=0.001):
return (f(x + h) - 2 * f(x) + f(x - h)) / h**2
for x in [-1, 0, 1]:
print(f"x = {x:>2}: f''(x) ≈ {round(second_derivative(f, x), 2) + 0.0}")The Hessian: second derivatives in many dimensions
A function of two variables has four second derivatives: differentiate by x or y, then by x or y again. Arranged in a grid, they form the Hessian matrix.
The two "mixed" entries in the corners are always equal for smooth functions, so the Hessian is symmetric.
import sympy as sp
x, y = sp.symbols("x y")
print("bowl x² + 4y²:", sp.hessian(x**2 + 4 * y**2, (x, y)))
print("saddle x² - y²: ", sp.hessian(x**2 - y**2, (x, y)))
print("other x³ + xy²:", sp.hessian(x**3 + x * y**2, (x, y)))Reading the results:
- Bowl: curvature 2 along x and 8 along y, both positive. It bends up in every direction, so its flat point is a minimum. And it is 4 times steeper-sided along y: a stretched bowl.
- Saddle: curvature +2 along x but -2 along y. Up in one direction, down in the other: a saddle point, neither a minimum nor a maximum.
The Jacobian: many outputs
So far each function produced one number. A neural network layer produces many: it takes a vector in and gives a vector out. Its derivative is the Jacobian.
For F(x, y) = (x·y, x + y²), with two outputs:
J = [ ∂(xy)/∂x ∂(xy)/∂y ] = [ y x ]
[ ∂(x + y²)/∂x ∂(x + y²)/∂y ] [ 1 2y ]import numpy as np
import sympy as sp
# Exactly, with SymPy
x, y = sp.symbols("x y")
print(sp.Matrix([x * y, x + y**2]).jacobian([x, y]))
# Numerically at (2, 3): nudge one input at a time
def F(x, y):
return np.array([x * y, x + y**2])
h = 0.00001
column_x = (F(2 + h, 3) - F(2 - h, 3)) / (2 * h) # how the outputs change with x
column_y = (F(2, 3 + h) - F(2, 3 - h)) / (2 * h) # how the outputs change with y
print(np.round(np.column_stack([column_x, column_y]), 4) + 0.0)At (2, 3) the formula gives [[3, 2], [1, 6]], matching the numbers.
Common mistakes
Use the second derivative test to classify the flat points of a cubic.
def f(x):
return x**3 - 3 * x
h = 0.001
# f has flat points at x = -1 and x = 1.
# For each, estimate f''(x) with (f(x + h) - 2 * f(x) + f(x - h)) / h**2,
# round it to 2 places, and print "maximum" if it is negative or "minimum" if positive.
# Output must be exactly:
# x = -1: f'' = -6.0, maximum
# x = 1: f'' = 6.0, minimum
At a flat point of f(x, y), the Hessian has curvature +3 in one direction and -2 in another. What is the point?
Summary
- The second derivative measures curvature: positive bends up (bowl), negative bends down (hill).
- Second derivative test: a flat point with
f'' > 0is a minimum, withf'' < 0a maximum. - The Hessian is the matrix of second partial derivatives: it describes curvature in every direction and can reveal saddle points.
- The Jacobian is the matrix of first derivatives of a function with many outputs. For a linear layer it is the weight matrix.
- Backpropagation chains Jacobians. Optimisers care about curvature, but deep learning avoids building full Hessians.
Next: the chain rule module, starting with functions inside functions.
Finished reading? Tick it off to track your progress.