Velamuvi AIAcademy

Lesson 8 of 16 · Many Variables

The Jacobian and Hessian

Understand second derivatives and curvature, the second derivative test, the Hessian matrix of a function of many variables, and the Jacobian matrix of a function with many outputs, with NumPy and SymPy examples.

  • 10 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What a second derivative measures: curvature
  • How to tell a minimum from a maximum with the second derivative test
  • What the Hessian matrix is and what it says about a landscape
  • What the Jacobian matrix is, for functions with several outputs
  • Where Jacobians and Hessians appear in machine learning

The derivative of the derivative

The derivative f'(x) is itself a function, so you can differentiate it again. The result is the second derivative, f''(x).

For f(x) = x³ - 3x:

f'(x)  = 3x² - 3
f''(x) = 6x
f''(x)The curve...Shape
positivebends upwards, slope increasinglike a bowl ∪
negativebends downwards, slope decreasinglike a hill ∩
zerois not bending at that pointpossibly changing from one to the other

The second derivative test

A flat point, where f'(x) = 0, is called a critical point. The second derivative tells you what kind it is:

  • f'(x) = 0 and f''(x) > 0 → bottom of a bowl → minimum
  • f'(x) = 0 and f''(x) < 0 → top of a hill → maximum
  • f'(x) = 0 and f''(x) = 0 → the test cannot tell
-3-2-10123-4-2024maxmin
f(x) = x³ - 3x
at x = 1: f(x) = -2slope of tangent = 0
Move the point. The tangent is flat at x = -1 (top of a hill) and x = 1 (bottom of a valley).

A numerical second derivative uses three points:

Try it yourselfPython
def f(x):
    return x**3 - 3 * x

def second_derivative(f, x, h=0.001):
    return (f(x + h) - 2 * f(x) + f(x - h)) / h**2

for x in [-1, 0, 1]:
    print(f"x = {x:>2}: f''(x) ≈ {round(second_derivative(f, x), 2) + 0.0}")

The Hessian: second derivatives in many dimensions

A function of two variables has four second derivatives: differentiate by x or y, then by x or y again. Arranged in a grid, they form the Hessian matrix.

The two "mixed" entries in the corners are always equal for smooth functions, so the Hessian is symmetric.

Try it yourselfPython
import sympy as sp

x, y = sp.symbols("x y")

print("bowl   x² + 4y²:", sp.hessian(x**2 + 4 * y**2, (x, y)))
print("saddle x² - y²: ", sp.hessian(x**2 - y**2, (x, y)))
print("other  x³ + xy²:", sp.hessian(x**3 + x * y**2, (x, y)))

Reading the results:

  • Bowl: curvature 2 along x and 8 along y, both positive. It bends up in every direction, so its flat point is a minimum. And it is 4 times steeper-sided along y: a stretched bowl.
  • Saddle: curvature +2 along x but -2 along y. Up in one direction, down in the other: a saddle point, neither a minimum nor a maximum.

The Jacobian: many outputs

So far each function produced one number. A neural network layer produces many: it takes a vector in and gives a vector out. Its derivative is the Jacobian.

For F(x, y) = (x·y, x + y²), with two outputs:

J = [ ∂(xy)/∂x       ∂(xy)/∂y     ]   =   [ y   x  ]
    [ ∂(x + y²)/∂x   ∂(x + y²)/∂y ]       [ 1   2y ]
Try it yourselfPython
import numpy as np
import sympy as sp

# Exactly, with SymPy
x, y = sp.symbols("x y")
print(sp.Matrix([x * y, x + y**2]).jacobian([x, y]))

# Numerically at (2, 3): nudge one input at a time
def F(x, y):
    return np.array([x * y, x + y**2])

h = 0.00001
column_x = (F(2 + h, 3) - F(2 - h, 3)) / (2 * h)   # how the outputs change with x
column_y = (F(2, 3 + h) - F(2, 3 - h)) / (2 * h)   # how the outputs change with y
print(np.round(np.column_stack([column_x, column_y]), 4) + 0.0)

At (2, 3) the formula gives [[3, 2], [1, 6]], matching the numbers.

Common mistakes

Practice exercise

Use the second derivative test to classify the flat points of a cubic.

Your solutionPython
def f(x):
    return x**3 - 3 * x

h = 0.001

# f has flat points at x = -1 and x = 1.
# For each, estimate f''(x) with (f(x + h) - 2 * f(x) + f(x - h)) / h**2,
# round it to 2 places, and print "maximum" if it is negative or "minimum" if positive.

# Output must be exactly:
# x = -1: f'' = -6.0, maximum
# x = 1: f'' = 6.0, minimum
Knowledge check

At a flat point of f(x, y), the Hessian has curvature +3 in one direction and -2 in another. What is the point?

Summary

  • The second derivative measures curvature: positive bends up (bowl), negative bends down (hill).
  • Second derivative test: a flat point with f'' > 0 is a minimum, with f'' < 0 a maximum.
  • The Hessian is the matrix of second partial derivatives: it describes curvature in every direction and can reveal saddle points.
  • The Jacobian is the matrix of first derivatives of a function with many outputs. For a linear layer it is the weight matrix.
  • Backpropagation chains Jacobians. Optimisers care about curvature, but deep learning avoids building full Hessians.

Next: the chain rule module, starting with functions inside functions.

Finished reading? Tick it off to track your progress.