Lesson 12 of 16 · The Chain Rule
Automatic Differentiation
Understand automatic differentiation, the technique behind PyTorch, TensorFlow and JAX: forward mode with value-derivative pairs, reverse mode with a recorded tape, and why reverse mode trains neural networks. Build both in a few lines of Python.
- 7 min read
- Intermediate
- Reviewed
- What automatic differentiation is, and how it differs from numerical and symbolic methods
- How forward mode carries a value and a derivative together
- How reverse mode records operations and then runs backwards
- Why reverse mode is the right choice for neural networks
- What gradient code looks like in PyTorch
The best of both worlds
Earlier you met numerical derivatives (easy but slightly inaccurate and slow for many inputs) and symbolic derivatives (exact, but formulas swell). Automatic differentiation is exact and fast.
It does not manipulate formulas and does not use a small step h. It just knows the local derivative of each simple operation and combines them with the chain rule, exactly as you did by hand with computation graphs.
Forward mode: carry the derivative along
Idea: instead of a plain number, carry a pair: (value, derivative). Every operation updates both.
The rules come straight from the derivative rules:
(a, a') + (b, b') = (a + b, a' + b') sum rule
(a, a') × (b, b') = (a × b, a'·b + a·b') product rule
exp((a, a')) = (eᵃ, eᵃ · a') chain ruleimport math
def add(p, q):
return (p[0] + q[0], p[1] + q[1])
def mul(p, q):
return (p[0] * q[0], p[1] * q[0] + p[0] * q[1])
def exp(p):
return (math.exp(p[0]), math.exp(p[0]) * p[1])
# f(x) = x · eˣ at x = 1. Start with dx/dx = 1.
x = (1.0, 1.0)
result = mul(x, exp(x))
print("value:", round(result[0], 4))
print("derivative:", round(result[1], 4)) # by the product rule: eˣ + x·eˣ = 2e ≈ 5.4366One run of the program gives the value and the exact derivative. Constants are pairs with derivative 0, such as (2.0, 0.0).
Reverse mode: record, then go backwards
Forward mode gives the derivative with respect to one input per run. A neural network's loss has millions of inputs (its weights), so that would need millions of runs. Reverse mode records every operation on a list, called a tape, then walks it backwards once, exactly like the backward pass in the last lesson.
Here is a complete, tiny reverse-mode autodiff in plain Python:
values = [] # the value of every node
parents = [] # for every node: a list of (parent node, local derivative)
def var(v):
values.append(v)
parents.append([])
return len(values) - 1 # a node is just its position in the lists
def add(a, b):
values.append(values[a] + values[b])
parents.append([(a, 1.0), (b, 1.0)])
return len(values) - 1
def mul(a, b):
values.append(values[a] * values[b])
parents.append([(a, values[b]), (b, values[a])]) # the swap rule
return len(values) - 1
def backward(output):
grads = [0.0] * len(values)
grads[output] = 1.0
for node in range(output, -1, -1): # newest to oldest
for parent, local in parents[node]:
grads[parent] += grads[node] * local # chain rule, adding paths
return grads
# f = (x + y) · z, the graph from the last lesson
x = var(-2.0)
y = var(5.0)
z = var(-4.0)
f = mul(add(x, y), z)
grads = backward(f)
print("f =", values[f])
print("∂f/∂x =", grads[x], " ∂f/∂y =", grads[y], " ∂f/∂z =", grads[z])Twenty lines, and it computes exact gradients for any formula built from add and mul. Real libraries work the same way, with hundreds of operations and code that runs on GPUs.
What it looks like in PyTorch
PyTorch builds the tape for you whenever a tensor has requires_grad=True. This code is for reading only (PyTorch does not run in the browser):
import torch
w = torch.tensor(0.5, requires_grad=True) # "track operations on w"
b = torch.tensor(-0.5, requires_grad=True)
x, y = torch.tensor(2.0), torch.tensor(1.0)
p = torch.sigmoid(w * x + b) # forward pass, recorded on the tape
loss = (p - y) ** 2
loss.backward() # reverse-mode autodiff
print(w.grad, b.grad) # tensor(-0.3549) tensor(-0.1774)The value of w.grad matches the -0.3549 you computed with the chain rule two lessons ago. loss.backward() is the backward function above, at industrial scale.
Common mistakes
Use forward-mode automatic differentiation with value-derivative pairs to differentiate a polynomial exactly.
def add(p, q):
return (p[0] + q[0], p[1] + q[1])
def mul(p, q):
return (p[0] * q[0], p[1] * q[0] + p[0] * q[1])
# Use forward mode to compute f(x) = x * x * x + 2 * x at x = 3.
# Start with x = (3.0, 1.0) and the constant two = (2.0, 0.0).
# Print the value and the derivative.
# Output must be exactly:
# value: 33.0
# derivative: 29.0
Why do deep learning libraries use reverse-mode automatic differentiation?
Summary
- Automatic differentiation applies exact derivative rules to each operation as code runs: exact and fast.
- Forward mode carries (value, derivative) pairs: one run per input.
- Reverse mode records a tape and walks it backwards: one pass gives the gradient for every input.
- Backpropagation is reverse-mode autodiff, and it is what
loss.backward()does in PyTorch.
Next: the final module, where gradients are finally used to learn, starting with gradient descent.
Finished reading? Tick it off to track your progress.