Velamuvi AIAcademy

Lesson 9 of 16 · The Chain Rule

Composing Functions

Understand composite functions, f(g(x)): inner and outer functions, why order matters, and how a neuron, a layer and a whole neural network are functions inside functions, ready for the chain rule.

  • 7 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What a composite function is, and how to write it
  • How to spot the inner and outer functions
  • Why the order of composition matters
  • How a neuron is a composition of a linear step and an activation
  • How a whole neural network is one long composition

Functions inside functions

When the output of one function becomes the input of another, you have composed them.

Try it yourselfPython
def double(x):
    return 2 * x

def add_three(x):
    return x + 3

print(add_three(double(5)))   # first double: 10, then add 3: 13
print(double(add_three(5)))   # first add 3: 8, then double: 16

Read f(g(x)) from the inside out: the innermost function runs first.

Order matters

The two lines above gave 13 and 16. Composition is like putting on socks and shoes: the same two steps in a different order give a different result.

Learning to see this structure is the key skill for the next lesson: the chain rule differentiates compositions one layer at a time.

A neuron is a composition

An artificial neuron does two things in order:

  1. Linear step: z = w·x + b (weight times input, plus bias)
  2. Activation: a = σ(z), for example the sigmoid

So a neuron computes σ(w·x + b): the sigmoid after a linear function. Move the sliders to change the inner function and watch the S-shape shift and stretch:

a = σ(1x + 0)
-6-4-20246-0.200.20.40.60.811.2
σ(w·x + b)
The weight controls how steep the S is (and a negative weight flips it). The bias slides it left or right. The outer sigmoid always keeps the output between 0 and 1.
Try it yourselfPython
import math

def linear(x, w, b):
    return w * x + b

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def neuron(x, w, b):
    return sigmoid(linear(x, w, b))      # outer(inner(x))

for x in [-2, 0, 2]:
    print(f"x = {x:>2}: z = {linear(x, 2, -1):>2}, output = {neuron(x, 2, -1):.4f}")

A network is a long composition

A neural network stacks layers, and each layer is a function of the previous layer's output:

output = layer3(layer2(layer1(x)))
loss   = loss_function(layer3(layer2(layer1(x))), true_answer)
Try it yourselfPython
import numpy as np

def relu(z):
    return np.maximum(0, z)

W1 = np.array([[1.0, -1.0], [0.5, 2.0]])   # layer 1: 2 inputs -> 2 outputs
W2 = np.array([[1.0, 1.0]])                # layer 2: 2 inputs -> 1 output

def layer1(x):
    return relu(W1 @ x)

def layer2(h):
    return W2 @ h

x = np.array([2.0, 1.0])
h = layer1(x)
print("after layer 1:", h)
print("network output:", layer2(h))
print("same, in one line:", layer2(layer1(x)))

Even the biggest language models are, mathematically, one enormous composite function: text in, a few hundred layers composed together, probabilities out.

Common mistakes

Practice exercise

Compose functions in both orders, then compute a neuron as an activation after a linear step.

Your solutionPython
import math

def double(x):
    return 2 * x

def add_three(x):
    return x + 3

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

# 1. Print add_three(double(5)) and double(add_three(5)).
# 2. Print a neuron's output sigmoid(w * x + b) for x = 1, w = 2, b = -1,
#    rounded to 4 places.

# Output must be exactly:
# add_three(double(5)) = 13
# double(add_three(5)) = 16
# neuron output = 0.7311
Knowledge check

For h(x) = √(x² + 1), which are the inner and outer functions?

Summary

  • A composite function f(g(x)) applies g first, then f. Read from the inside out.
  • Order matters: f(g(x)) is usually different from g(f(x)).
  • A neuron is an activation after a linear step: σ(w·x + b).
  • A neural network, and its loss, is one long composition of layers.
  • Non-linear activations make composing layers worthwhile.

Next: the chain rule, which differentiates compositions.

Finished reading? Tick it off to track your progress.