Lesson 11 of 16 · The Chain Rule
Computation Graphs
See how a calculation becomes a computation graph, how the forward pass computes values, and how the backward pass applies the chain rule node by node to get every gradient: the exact procedure behind backpropagation.
- 9 min read
- Intermediate
- Reviewed
- What a computation graph is
- How the forward pass computes the output
- How the backward pass computes every gradient with the chain rule
- The simple gradient rules for add, multiply and max nodes
- Why gradients add up when a value is used twice
A calculation as a picture
Any formula can be broken into small steps, each one simple operation. Drawn as boxes and arrows, those steps form a computation graph.
Take f = (x + y) · z with x = -2, y = 5, z = -4. Give the middle result a name, q:
x = -2 ──┐
(+) ── q = 3 ──┐
y = 5 ──┘ (×) ── f = -12
z = -4 ─────────────────┘The forward pass
q = x + y = -2 + 5 = 3
f = q · z = 3 · (-4) = -12The backward pass
Now we want the gradient: ∂f/∂x, ∂f/∂y and ∂f/∂z. Instead of deriving a formula for each, we walk the graph backwards, from the output to the inputs, applying the chain rule at each node.
Step by step:
- Start at the top:
∂f/∂f = 1. - Multiply node,
f = q · z. Local derivatives:∂f/∂q = z = -4and∂f/∂z = q = 3. So ∂f/∂z = 3, and the gradient arriving at q is -4. - Add node,
q = x + y. Local derivatives are both 1, so the gradient passes through unchanged: ∂f/∂x = -4 · 1 = -4 and ∂f/∂y = -4.
# Forward pass
x, y, z = -2.0, 5.0, -4.0
q = x + y
f = q * z
print("f =", f)
# Backward pass: chain rule, from the output back
df_df = 1.0
df_dq = df_df * z # multiply node: local derivative wrt q is z
df_dz = df_df * q # ... and wrt z is q
df_dx = df_dq * 1.0 # add node: local derivative is 1
df_dy = df_dq * 1.0
print(f"∂f/∂x = {df_dx}, ∂f/∂y = {df_dy}, ∂f/∂z = {df_dz}")Check one: if x goes up by 1, q goes up by 1, and f = q·z changes by 1·(-4) = -4. Correct.
Three simple patterns
Each kind of node handles gradients in a predictable way:
| Node | Forward | Backward (what each input receives) |
|---|---|---|
add a + b | adds | the same gradient, copied to both inputs |
multiply a · b | multiplies | input a gets gradient × b, input b gets gradient × a (they swap) |
max max(a, b) | picks the bigger | the bigger input gets all the gradient, the other gets 0 |
ReLU is a max node, max(0, z): the gradient flows through when z is positive and is blocked when it is negative.
When a value is used twice
If a value feeds into two places, it affects the output along two paths, and its gradient is the sum of the gradients from both paths.
This is the multivariable chain rule: add up the contributions of every path from the input to the output.
A neuron as a graph
x1 ─(×w1)─┐
(+) ── (+b) ── z ── ReLU ── a ── (a - y)² ── L
x2 ─(×w2)─┘x1, x2, y = 1.0, 2.0, 3.0 # inputs and the correct answer
w1, w2, b = 0.5, 1.0, 0.2 # parameters
# Forward
z = w1 * x1 + w2 * x2 + b # 0.5 + 2.0 + 0.2 = 2.7
a = max(0.0, z) # ReLU
L = (a - y) ** 2
print(f"z = {z:.1f}, a = {a:.1f}, loss = {L:.2f}")
# Backward
dL_da = 2 * (a - y) # from (a - y)²
if z > 0: # ReLU passes the gradient only if z > 0
da_dz = 1.0
else:
da_dz = 0.0
dL_dz = dL_da * da_dz
dL_dw1 = dL_dz * x1 # multiply node: swap in the other input
dL_dw2 = dL_dz * x2
dL_db = dL_dz # add node: copy the gradient
print(f"∂L/∂w1 = {dL_dw1:.2f}, ∂L/∂w2 = {dL_dw2:.2f}, ∂L/∂b = {dL_db:.2f}")All three gradients are negative: the neuron's output (2.7) is too small, so every parameter should increase a little. Notice that w2 gets twice the gradient of w1, because its input x2 is twice as big.
Common mistakes
Run a forward pass and a backward pass by hand through a three-node graph.
a, b, c = 2.0, 3.0, -4.0 # f = (a * b + c) ** 2, computed in steps: # u = a * b, v = u + c, f = v ** 2 # 1. Forward pass: compute u, v and f. # 2. Backward pass: df/dv = 2 * v, then pass it back through the add node # and the multiply node to get df/da, df/db and df/dc. # Output must be exactly: # f = 4.0 # df/da = 12.0 # df/db = 8.0 # df/dc = 4.0
In a computation graph, a multiply node computes q = a · b with a = 4 and b = 5. The gradient arriving from above is 2. What gradient does a receive?
Summary
- A computation graph breaks a calculation into simple operations.
- The forward pass computes values from inputs to output.
- The backward pass starts with 1 at the output and multiplies by local derivatives on the way back: the chain rule.
- Add nodes copy the gradient, multiply nodes swap in the other input, max nodes route it to the winner.
- If a value is used more than once, its gradients add up.
Next: how libraries automate all of this.
Finished reading? Tick it off to track your progress.