Velamuvi AIAcademy

Lesson 3 of 16 · Change

Derivative Rules

Learn the derivative rules that give answers instantly: constant, power, constant multiple, sum, exponential and logarithm, plus the product and quotient rules and the derivatives of sigmoid, ReLU and squared error used in machine learning.

  • 7 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • The constant, power, constant multiple and sum rules
  • The derivatives of eˣ and ln(x)
  • How to differentiate any polynomial in one line
  • The derivatives of sigmoid, ReLU and squared error
  • How to check a derivative rule with code

Why rules?

In the last lesson you found the derivative of x² with a limit, and it took several lines of algebra. Mathematicians have done that work once for the common building blocks, giving rules you can apply instantly.

The four basic rules

RuleFunctionDerivativeExample
Constantc0d/dx 7 = 0
Powerxⁿn·xⁿ⁻¹d/dx x³ = 3x²
Constant multiplec·f(x)c·f'(x)d/dx 5x³ = 15x²
Sumf(x) + g(x)f'(x) + g'(x)d/dx (x² + x) = 2x + 1

The power rule works for any power n, including negative ones and fractions:

  • x = x¹ → 1·x⁰ = 1 (a line y = x has slope 1)
  • x² → 2x
  • 1/x = x⁻¹ → -1·x⁻² = -1/x²
  • √x = x^0.5 → 0.5·x^(-0.5) = 1/(2√x)

A constant has derivative 0 because a flat line has no slope. That is also why adding a constant to a function (moving it up) does not change its slopes.

Checking a rule with code

Never trust, always verify. Compare the rule with a numerical estimate:

Try it yourselfPython
def f(x):
    return 3 * x**4 - 5 * x**2 + 7 * x - 2

def f_prime(x):                       # from the rules
    return 12 * x**3 - 10 * x + 7

h = 0.00001
for x in [-1, 0, 1, 2]:
    estimate = (f(x + h) - f(x - h)) / (2 * h)   # a very accurate slope estimate
    print(f"x = {x:>2}: rule {f_prime(x):>4}, estimate {estimate:.3f}")

The estimate uses two points either side of x, called a central difference. It is more accurate than the one-sided version from the last lesson.

Exponentials and logarithms

FunctionDerivativeWhy it matters
eˣeˣthe only function that is its own derivative
e^(kx)k·e^(kx)growth and decay at rate k
ln(x)1/xlog-loss and likelihoods in ML

eˣ is special: at every point its slope equals its value. That is the reason e ≈ 2.718 is used everywhere in maths and AI.

Derivatives you will meet in machine learning

Squared error. For one example with true value y and prediction p, the error is (p - y)². Its derivative with respect to the prediction p is 2(p - y). If the prediction is too high, this is positive, which says "decrease p".

ReLU. ReLU(x) = max(0, x) has slope 0 for negative x and slope 1 for positive x. (At exactly 0 it has a corner, and libraries simply use 0 or 1.)

Sigmoid. σ(x) = 1/(1 + e⁻ˣ) has a beautifully simple derivative:

σ'(x) = σ(x) · (1 - σ(x))
-6-4-2024600.20.40.60.81
σ(x)σ'(x) = σ(x)(1 - σ(x))
The sigmoid is steepest at x = 0, where its slope is 0.25. Far from 0 it flattens out and its slope almost vanishes.
Try it yourselfPython
import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

h = 0.00001
for x in [-4, 0, 4]:
    rule = sigmoid(x) * (1 - sigmoid(x))
    estimate = (sigmoid(x + h) - sigmoid(x - h)) / (2 * h)
    print(f"x = {x:>2}: rule {rule:.4f}, estimate {estimate:.4f}")

Notice that the slope is at most 0.25 and is tiny for large inputs. This fact causes the "vanishing gradient" problem in deep networks, which you will meet in the chain rule lesson.

Common mistakes

Practice exercise

Differentiate a polynomial with the rules, then check your answer numerically.

Your solutionPython
def f(x):
    return 4 * x**3 - 2 * x + 1

# 1. Write f_prime(x) using the derivative rules.
# 2. For x = 0, 1, 2 print the rule's answer and a central-difference estimate
#    (f(x + h) - f(x - h)) / (2 * h) with h = 0.00001, rounded to 3 places.

# Output must be exactly:
# x = 0: rule -2, estimate -2.0
# x = 1: rule 10, estimate 10.0
# x = 2: rule 46, estimate 46.0
Knowledge check

What is the derivative of f(x) = 5x³ + 2x - 9?

Summary

  • Constant → 0. Power: xⁿ → n·xⁿ⁻¹. Constant multiple and sum: work term by term.
  • eˣ is its own derivative, and ln(x) has derivative 1/x.
  • In ML: squared error (p - y)² → 2(p - y), ReLU → 0 or 1, sigmoid → σ(1 - σ).
  • Check any derivative with a central difference: (f(x + h) - f(x - h)) / (2h).

Next: three ways to get derivatives in Python.

Finished reading? Tick it off to track your progress.