Lesson 9 of 16 · The Chain Rule
Composing Functions
Understand composite functions, f(g(x)): inner and outer functions, why order matters, and how a neuron, a layer and a whole neural network are functions inside functions, ready for the chain rule.
- 7 min read
- Intermediate
- Reviewed
- What a composite function is, and how to write it
- How to spot the inner and outer functions
- Why the order of composition matters
- How a neuron is a composition of a linear step and an activation
- How a whole neural network is one long composition
Functions inside functions
When the output of one function becomes the input of another, you have composed them.
def double(x):
return 2 * x
def add_three(x):
return x + 3
print(add_three(double(5))) # first double: 10, then add 3: 13
print(double(add_three(5))) # first add 3: 8, then double: 16Read f(g(x)) from the inside out: the innermost function runs first.
Order matters
The two lines above gave 13 and 16. Composition is like putting on socks and shoes: the same two steps in a different order give a different result.
Learning to see this structure is the key skill for the next lesson: the chain rule differentiates compositions one layer at a time.
A neuron is a composition
An artificial neuron does two things in order:
- Linear step: z = w·x + b (weight times input, plus bias)
- Activation: a = σ(z), for example the sigmoid
So a neuron computes σ(w·x + b): the sigmoid after a linear function. Move the sliders to change the inner function and watch the S-shape shift and stretch:
import math
def linear(x, w, b):
return w * x + b
def sigmoid(z):
return 1 / (1 + math.exp(-z))
def neuron(x, w, b):
return sigmoid(linear(x, w, b)) # outer(inner(x))
for x in [-2, 0, 2]:
print(f"x = {x:>2}: z = {linear(x, 2, -1):>2}, output = {neuron(x, 2, -1):.4f}")A network is a long composition
A neural network stacks layers, and each layer is a function of the previous layer's output:
output = layer3(layer2(layer1(x)))
loss = loss_function(layer3(layer2(layer1(x))), true_answer)import numpy as np
def relu(z):
return np.maximum(0, z)
W1 = np.array([[1.0, -1.0], [0.5, 2.0]]) # layer 1: 2 inputs -> 2 outputs
W2 = np.array([[1.0, 1.0]]) # layer 2: 2 inputs -> 1 output
def layer1(x):
return relu(W1 @ x)
def layer2(h):
return W2 @ h
x = np.array([2.0, 1.0])
h = layer1(x)
print("after layer 1:", h)
print("network output:", layer2(h))
print("same, in one line:", layer2(layer1(x)))Even the biggest language models are, mathematically, one enormous composite function: text in, a few hundred layers composed together, probabilities out.
Common mistakes
Compose functions in both orders, then compute a neuron as an activation after a linear step.
import math
def double(x):
return 2 * x
def add_three(x):
return x + 3
def sigmoid(z):
return 1 / (1 + math.exp(-z))
# 1. Print add_three(double(5)) and double(add_three(5)).
# 2. Print a neuron's output sigmoid(w * x + b) for x = 1, w = 2, b = -1,
# rounded to 4 places.
# Output must be exactly:
# add_three(double(5)) = 13
# double(add_three(5)) = 16
# neuron output = 0.7311
For h(x) = √(x² + 1), which are the inner and outer functions?
Summary
- A composite function f(g(x)) applies g first, then f. Read from the inside out.
- Order matters: f(g(x)) is usually different from g(f(x)).
- A neuron is an activation after a linear step: σ(w·x + b).
- A neural network, and its loss, is one long composition of layers.
- Non-linear activations make composing layers worthwhile.
Next: the chain rule, which differentiates compositions.
Finished reading? Tick it off to track your progress.