Velamuvi AIAcademy

Lesson 11 of 16 · The Chain Rule

Computation Graphs

See how a calculation becomes a computation graph, how the forward pass computes values, and how the backward pass applies the chain rule node by node to get every gradient: the exact procedure behind backpropagation.

  • 9 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What a computation graph is
  • How the forward pass computes the output
  • How the backward pass computes every gradient with the chain rule
  • The simple gradient rules for add, multiply and max nodes
  • Why gradients add up when a value is used twice

A calculation as a picture

Any formula can be broken into small steps, each one simple operation. Drawn as boxes and arrows, those steps form a computation graph.

Take f = (x + y) · z with x = -2, y = 5, z = -4. Give the middle result a name, q:

  x = -2 ──┐
           (+) ── q = 3 ──┐
  y =  5 ──┘              (×) ── f = -12
  z = -4 ─────────────────┘

The forward pass

q = x + y = -2 + 5 = 3
f = q · z = 3 · (-4) = -12

The backward pass

Now we want the gradient: ∂f/∂x, ∂f/∂y and ∂f/∂z. Instead of deriving a formula for each, we walk the graph backwards, from the output to the inputs, applying the chain rule at each node.

Step by step:

  1. Start at the top: ∂f/∂f = 1.
  2. Multiply node, f = q · z. Local derivatives: ∂f/∂q = z = -4 and ∂f/∂z = q = 3. So ∂f/∂z = 3, and the gradient arriving at q is -4.
  3. Add node, q = x + y. Local derivatives are both 1, so the gradient passes through unchanged: ∂f/∂x = -4 · 1 = -4 and ∂f/∂y = -4.
Try it yourselfPython
# Forward pass
x, y, z = -2.0, 5.0, -4.0
q = x + y
f = q * z
print("f =", f)

# Backward pass: chain rule, from the output back
df_df = 1.0
df_dq = df_df * z          # multiply node: local derivative wrt q is z
df_dz = df_df * q          # ... and wrt z is q
df_dx = df_dq * 1.0        # add node: local derivative is 1
df_dy = df_dq * 1.0

print(f"∂f/∂x = {df_dx}, ∂f/∂y = {df_dy}, ∂f/∂z = {df_dz}")

Check one: if x goes up by 1, q goes up by 1, and f = q·z changes by 1·(-4) = -4. Correct.

Three simple patterns

Each kind of node handles gradients in a predictable way:

NodeForwardBackward (what each input receives)
add a + baddsthe same gradient, copied to both inputs
multiply a · bmultipliesinput a gets gradient × b, input b gets gradient × a (they swap)
max max(a, b)picks the biggerthe bigger input gets all the gradient, the other gets 0

ReLU is a max node, max(0, z): the gradient flows through when z is positive and is blocked when it is negative.

When a value is used twice

If a value feeds into two places, it affects the output along two paths, and its gradient is the sum of the gradients from both paths.

This is the multivariable chain rule: add up the contributions of every path from the input to the output.

A neuron as a graph

x1 ─(×w1)─┐
          (+) ── (+b) ── z ── ReLU ── a ── (a - y)² ── L
x2 ─(×w2)─┘
Try it yourselfPython
x1, x2, y = 1.0, 2.0, 3.0          # inputs and the correct answer
w1, w2, b = 0.5, 1.0, 0.2          # parameters

# Forward
z = w1 * x1 + w2 * x2 + b          # 0.5 + 2.0 + 0.2 = 2.7
a = max(0.0, z)                    # ReLU
L = (a - y) ** 2
print(f"z = {z:.1f}, a = {a:.1f}, loss = {L:.2f}")

# Backward
dL_da = 2 * (a - y)                # from (a - y)²
if z > 0:                          # ReLU passes the gradient only if z > 0
    da_dz = 1.0
else:
    da_dz = 0.0
dL_dz = dL_da * da_dz
dL_dw1 = dL_dz * x1                # multiply node: swap in the other input
dL_dw2 = dL_dz * x2
dL_db = dL_dz                      # add node: copy the gradient
print(f"∂L/∂w1 = {dL_dw1:.2f}, ∂L/∂w2 = {dL_dw2:.2f}, ∂L/∂b = {dL_db:.2f}")

All three gradients are negative: the neuron's output (2.7) is too small, so every parameter should increase a little. Notice that w2 gets twice the gradient of w1, because its input x2 is twice as big.

Common mistakes

Practice exercise

Run a forward pass and a backward pass by hand through a three-node graph.

Your solutionPython
a, b, c = 2.0, 3.0, -4.0

# f = (a * b + c) ** 2, computed in steps:
#   u = a * b,   v = u + c,   f = v ** 2
# 1. Forward pass: compute u, v and f.
# 2. Backward pass: df/dv = 2 * v, then pass it back through the add node
#    and the multiply node to get df/da, df/db and df/dc.

# Output must be exactly:
# f = 4.0
# df/da = 12.0
# df/db = 8.0
# df/dc = 4.0
Knowledge check

In a computation graph, a multiply node computes q = a · b with a = 4 and b = 5. The gradient arriving from above is 2. What gradient does a receive?

Summary

  • A computation graph breaks a calculation into simple operations.
  • The forward pass computes values from inputs to output.
  • The backward pass starts with 1 at the output and multiplies by local derivatives on the way back: the chain rule.
  • Add nodes copy the gradient, multiply nodes swap in the other input, max nodes route it to the winner.
  • If a value is used more than once, its gradients add up.

Next: how libraries automate all of this.

Finished reading? Tick it off to track your progress.