Lesson 10 of 16 · The Chain Rule
The Chain Rule
Learn the chain rule, the most important derivative rule in machine learning: differentiate functions inside functions by multiplying rates of change. Worked examples, a neuron's gradient, and why gradients vanish in deep networks.
- 8 min read
- Intermediate
- Reviewed
- What the chain rule says, in words and in notation
- How to differentiate a composite function step by step
- How to compute the gradient of a neuron's loss with the chain rule
- Why long chains can make gradients vanish
- How the chain rule leads to backpropagation
Rates multiply
Imagine three connected gears. Gear A turns 3 times for every turn of gear B, and gear B turns 2 times for every turn of gear C. How many times does A turn for each turn of C?
3 × 2 = 6. Rates of change along a chain multiply. That is the whole chain rule.
Writing u = g(x) for the inner value, the same rule in Leibniz notation looks like fractions cancelling:
dy dy du
── = ── · ──
dx du dx"How y changes with x" = "how y changes with u" × "how u changes with x".
Step by step
More examples:
| Function | Inner u | Outer | Derivative |
|---|---|---|---|
(3x + 1)² | 3x + 1 | u² | 2(3x + 1) · 3 |
e^(-x²) | -x² | eᵘ | e^(-x²) · (-2x) |
ln(1 + x²) | 1 + x² | ln u | (1/(1 + x²)) · 2x |
σ(wx + b) (by w) | wx + b | σ(u) | σ'(wx + b) · x |
import math
def derivative(f, x, h=0.00001):
return (f(x + h) - f(x - h)) / (2 * h)
def h(x):
return math.exp(-x**2)
def h_prime(x): # from the chain rule
return math.exp(-x**2) * (-2 * x)
for x in [-1, 0.5, 2]:
print(f"x = {x:>4}: chain rule {h_prime(x):.5f}, numerical {derivative(h, x):.5f}")The gradient of a neuron
Now the real thing. A neuron predicts p = σ(w·x + b), and its loss for one example with true answer y is L = (p - y)². How does the loss change with the weight w?
The chain has three links: w affects z, z affects p, p affects L.
z = w·x + b → dz/dw = x
p = σ(z) → dp/dz = σ(z)(1 - σ(z)) = p(1 - p)
L = (p - y)² → dL/dp = 2(p - y)
dL/dw = dL/dp · dp/dz · dz/dw = 2(p - y) · p(1 - p) · ximport math
def sigmoid(z):
return 1 / (1 + math.exp(-z))
x, y = 2.0, 1.0 # one training example: input 2, correct answer 1
w, b = 0.5, -0.5 # current parameters
z = w * x + b
p = sigmoid(z)
L = (p - y) ** 2
dL_dp = 2 * (p - y)
dp_dz = p * (1 - p)
dz_dw = x
dL_dw = dL_dp * dp_dz * dz_dw
print(f"prediction {p:.4f}, loss {L:.4f}")
print(f"links: {dL_dp:.4f} × {dp_dz:.4f} × {dz_dw:.1f} = {dL_dw:.4f}")The gradient is negative, so increasing w lowers the loss: the prediction (0.62) is below the answer (1), and a bigger weight pushes it up. Every weight in every network gets its gradient this way.
Long chains and vanishing gradients
A deep network is a long chain, so its gradients are products of many local derivatives. That can go badly wrong.
The sigmoid's slope is at most 0.25. Chain 10 sigmoids and the gradient shrinks by up to:
for layers in [1, 5, 10, 20]:
print(f"{layers:>2} sigmoid layers: gradient multiplied by at most {0.25 ** layers:.2e}")Common mistakes
Compute the gradient of a neuron's loss with the chain rule, then confirm it numerically.
import math
def sigmoid(z):
return 1 / (1 + math.exp(-z))
x, y = 2.0, 1.0
w, b = 0.5, -0.5
# 1. Forward: z = w * x + b, p = sigmoid(z), loss = (p - y) ** 2
# 2. Chain rule: dL/dw = 2 * (p - y) * p * (1 - p) * x
# 3. Check it numerically with a central difference on w (h = 0.00001)
# Print each value rounded to 4 places.
# Output must be exactly:
# loss: 0.1425
# chain rule: -0.3549
# numerical: -0.3549
If y = sin(u) and u = 5x, what is dy/dx?
Summary
- Along a chain, rates of change multiply: dy/dx = dy/du · du/dx.
- Differentiate the outer function at the inner value, then multiply by the inner derivative.
- A neuron's gradient is a product of three links:
2(p - y) · p(1 - p) · x. - Long products of small numbers cause vanishing gradients. ReLU and residual connections help.
- Backpropagation is the chain rule, organised to run backwards through a network.
Next: computation graphs, the map that backpropagation follows.
Finished reading? Tick it off to track your progress.