Velamuvi AIAcademy

Lesson 10 of 16 · The Chain Rule

The Chain Rule

Learn the chain rule, the most important derivative rule in machine learning: differentiate functions inside functions by multiplying rates of change. Worked examples, a neuron's gradient, and why gradients vanish in deep networks.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What the chain rule says, in words and in notation
  • How to differentiate a composite function step by step
  • How to compute the gradient of a neuron's loss with the chain rule
  • Why long chains can make gradients vanish
  • How the chain rule leads to backpropagation

Rates multiply

Imagine three connected gears. Gear A turns 3 times for every turn of gear B, and gear B turns 2 times for every turn of gear C. How many times does A turn for each turn of C?

3 × 2 = 6. Rates of change along a chain multiply. That is the whole chain rule.

Writing u = g(x) for the inner value, the same rule in Leibniz notation looks like fractions cancelling:

dy     dy     du
──  =  ──  ·  ──
dx     du     dx

"How y changes with x" = "how y changes with u" × "how u changes with x".

Step by step

-2-1.5-1-0.500.511.505101520
h(x) = (3x + 1)²
at x = 0: f(x) = 1slope of tangent = 6
The slope shown always equals 6(3x + 1). At x = 0 it is 6, and at x = -1/3, where the inner function is 0, it is 0.

More examples:

FunctionInner uOuterDerivative
(3x + 1)²3x + 1u²2(3x + 1) · 3
e^(-x²)-x²eᵘe^(-x²) · (-2x)
ln(1 + x²)1 + x²ln u(1/(1 + x²)) · 2x
σ(wx + b) (by w)wx + bσ(u)σ'(wx + b) · x
Try it yourselfPython
import math

def derivative(f, x, h=0.00001):
    return (f(x + h) - f(x - h)) / (2 * h)

def h(x):
    return math.exp(-x**2)

def h_prime(x):                          # from the chain rule
    return math.exp(-x**2) * (-2 * x)

for x in [-1, 0.5, 2]:
    print(f"x = {x:>4}: chain rule {h_prime(x):.5f}, numerical {derivative(h, x):.5f}")

The gradient of a neuron

Now the real thing. A neuron predicts p = σ(w·x + b), and its loss for one example with true answer y is L = (p - y)². How does the loss change with the weight w?

The chain has three links: w affects z, z affects p, p affects L.

z = w·x + b      →   dz/dw = x
p = σ(z)         →   dp/dz = σ(z)(1 - σ(z)) = p(1 - p)
L = (p - y)²     →   dL/dp = 2(p - y)

dL/dw = dL/dp · dp/dz · dz/dw = 2(p - y) · p(1 - p) · x
Try it yourselfPython
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

x, y = 2.0, 1.0          # one training example: input 2, correct answer 1
w, b = 0.5, -0.5         # current parameters

z = w * x + b
p = sigmoid(z)
L = (p - y) ** 2

dL_dp = 2 * (p - y)
dp_dz = p * (1 - p)
dz_dw = x
dL_dw = dL_dp * dp_dz * dz_dw

print(f"prediction {p:.4f}, loss {L:.4f}")
print(f"links: {dL_dp:.4f} × {dp_dz:.4f} × {dz_dw:.1f} = {dL_dw:.4f}")

The gradient is negative, so increasing w lowers the loss: the prediction (0.62) is below the answer (1), and a bigger weight pushes it up. Every weight in every network gets its gradient this way.

Long chains and vanishing gradients

A deep network is a long chain, so its gradients are products of many local derivatives. That can go badly wrong.

The sigmoid's slope is at most 0.25. Chain 10 sigmoids and the gradient shrinks by up to:

Try it yourselfPython
for layers in [1, 5, 10, 20]:
    print(f"{layers:>2} sigmoid layers: gradient multiplied by at most {0.25 ** layers:.2e}")

Common mistakes

Practice exercise

Compute the gradient of a neuron's loss with the chain rule, then confirm it numerically.

Your solutionPython
import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

x, y = 2.0, 1.0
w, b = 0.5, -0.5

# 1. Forward: z = w * x + b, p = sigmoid(z), loss = (p - y) ** 2
# 2. Chain rule: dL/dw = 2 * (p - y) * p * (1 - p) * x
# 3. Check it numerically with a central difference on w (h = 0.00001)
# Print each value rounded to 4 places.

# Output must be exactly:
# loss: 0.1425
# chain rule: -0.3549
# numerical: -0.3549
Knowledge check

If y = sin(u) and u = 5x, what is dy/dx?

Summary

  • Along a chain, rates of change multiply: dy/dx = dy/du · du/dx.
  • Differentiate the outer function at the inner value, then multiply by the inner derivative.
  • A neuron's gradient is a product of three links: 2(p - y) · p(1 - p) · x.
  • Long products of small numbers cause vanishing gradients. ReLU and residual connections help.
  • Backpropagation is the chain rule, organised to run backwards through a network.

Next: computation graphs, the map that backpropagation follows.

Finished reading? Tick it off to track your progress.