Lesson 3 of 16 · Change
Derivative Rules
Learn the derivative rules that give answers instantly: constant, power, constant multiple, sum, exponential and logarithm, plus the product and quotient rules and the derivatives of sigmoid, ReLU and squared error used in machine learning.
- 7 min read
- Intermediate
- Reviewed
- The constant, power, constant multiple and sum rules
- The derivatives of eˣ and ln(x)
- How to differentiate any polynomial in one line
- The derivatives of sigmoid, ReLU and squared error
- How to check a derivative rule with code
Why rules?
In the last lesson you found the derivative of x² with a limit, and it took several lines of algebra. Mathematicians have done that work once for the common building blocks, giving rules you can apply instantly.
The four basic rules
| Rule | Function | Derivative | Example |
|---|---|---|---|
| Constant | c | 0 | d/dx 7 = 0 |
| Power | xⁿ | n·xⁿ⁻¹ | d/dx x³ = 3x² |
| Constant multiple | c·f(x) | c·f'(x) | d/dx 5x³ = 15x² |
| Sum | f(x) + g(x) | f'(x) + g'(x) | d/dx (x² + x) = 2x + 1 |
The power rule works for any power n, including negative ones and fractions:
x = x¹→1·x⁰ = 1(a line y = x has slope 1)x²→2x1/x = x⁻¹→-1·x⁻² = -1/x²√x = x^0.5→0.5·x^(-0.5) = 1/(2√x)
A constant has derivative 0 because a flat line has no slope. That is also why adding a constant to a function (moving it up) does not change its slopes.
Checking a rule with code
Never trust, always verify. Compare the rule with a numerical estimate:
def f(x):
return 3 * x**4 - 5 * x**2 + 7 * x - 2
def f_prime(x): # from the rules
return 12 * x**3 - 10 * x + 7
h = 0.00001
for x in [-1, 0, 1, 2]:
estimate = (f(x + h) - f(x - h)) / (2 * h) # a very accurate slope estimate
print(f"x = {x:>2}: rule {f_prime(x):>4}, estimate {estimate:.3f}")The estimate uses two points either side of x, called a central difference. It is more accurate than the one-sided version from the last lesson.
Exponentials and logarithms
| Function | Derivative | Why it matters |
|---|---|---|
eˣ | eˣ | the only function that is its own derivative |
e^(kx) | k·e^(kx) | growth and decay at rate k |
ln(x) | 1/x | log-loss and likelihoods in ML |
eˣ is special: at every point its slope equals its value. That is the reason e ≈ 2.718 is used everywhere in maths and AI.
Derivatives you will meet in machine learning
Squared error. For one example with true value y and prediction p, the error is (p - y)². Its derivative with respect to the prediction p is 2(p - y). If the prediction is too high, this is positive, which says "decrease p".
ReLU. ReLU(x) = max(0, x) has slope 0 for negative x and slope 1 for positive x. (At exactly 0 it has a corner, and libraries simply use 0 or 1.)
Sigmoid. σ(x) = 1/(1 + e⁻ˣ) has a beautifully simple derivative:
σ'(x) = σ(x) · (1 - σ(x))import math
def sigmoid(x):
return 1 / (1 + math.exp(-x))
h = 0.00001
for x in [-4, 0, 4]:
rule = sigmoid(x) * (1 - sigmoid(x))
estimate = (sigmoid(x + h) - sigmoid(x - h)) / (2 * h)
print(f"x = {x:>2}: rule {rule:.4f}, estimate {estimate:.4f}")Notice that the slope is at most 0.25 and is tiny for large inputs. This fact causes the "vanishing gradient" problem in deep networks, which you will meet in the chain rule lesson.
Common mistakes
Differentiate a polynomial with the rules, then check your answer numerically.
def f(x):
return 4 * x**3 - 2 * x + 1
# 1. Write f_prime(x) using the derivative rules.
# 2. For x = 0, 1, 2 print the rule's answer and a central-difference estimate
# (f(x + h) - f(x - h)) / (2 * h) with h = 0.00001, rounded to 3 places.
# Output must be exactly:
# x = 0: rule -2, estimate -2.0
# x = 1: rule 10, estimate 10.0
# x = 2: rule 46, estimate 46.0
What is the derivative of f(x) = 5x³ + 2x - 9?
Summary
- Constant → 0. Power: xⁿ → n·xⁿ⁻¹. Constant multiple and sum: work term by term.
eˣis its own derivative, andln(x)has derivative1/x.- In ML: squared error
(p - y)²→2(p - y), ReLU → 0 or 1, sigmoid →σ(1 - σ). - Check any derivative with a central difference:
(f(x + h) - f(x - h)) / (2h).
Next: three ways to get derivatives in Python.
Finished reading? Tick it off to track your progress.