Velamuvi AIAcademy

Lesson 6 of 16 · Matrices

Matrix Multiplication

Learn matrix-vector and matrix-matrix multiplication as rows dotted with columns, the shape rule, why order matters, and how one matrix multiplication makes predictions for a whole dataset.

  • 10 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • How to multiply a matrix by a vector
  • How to multiply two matrices, entry by entry
  • The shape rule that says when multiplication is possible
  • Why A × B is usually different from B × A
  • How one multiplication predicts for a whole dataset, or runs a neural network layer

Matrix times vector

You already know the dot product. Matrix multiplication is just many dot products at once.

[ 2  1 ]   [ 3 ]     [ 2×3 + 1×4 ]     [ 10 ]
[ 0  5 ] × [ 4 ]  =  [ 0×3 + 5×4 ]  =  [ 20 ]
[ 1  1 ]             [ 1×3 + 1×4 ]     [  7 ]
Try it yourselfPython
import numpy as np

A = np.array([
    [2, 1],
    [0, 5],
    [1, 1],
])
x = np.array([3, 4])

print(A @ x)

In NumPy, @ is the matrix multiplication operator.

Predictions for a whole dataset

Here is why this matters. Put every example in the rows of a data matrix X, and the model's weights in a vector w. Then X @ w takes the dot product of every example with the weights: all the predictions in one step.

Try it yourselfPython
import numpy as np

# Each row is a house: [size m², bedrooms, age]
X = np.array([
    [120, 3, 15],
    [ 80, 2, 30],
    [200, 5,  2],
])
w = np.array([40, 150, -20])     # weights from a trained model (thousands per unit)
b = 500                          # bias

predictions = X @ w + b
print(predictions)

Three houses, three predictions, no loop. With a million houses, the code is exactly the same.

Matrix times matrix

Try it yourselfPython
import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

print(A @ B)

The shape rule

You can only multiply when the shapes fit together.

(m × n) @ (n × p)  →  (m × p)
     └─same─┘
A shapeB shapeAllowed?Result shape
2 × 33 × 4yes2 × 4
100 × 2020 × 1yes100 × 1
2 × 32 × 3no: 3 ≠ 2
Try it yourselfPython
import numpy as np

A = np.ones((2, 3))
B = np.ones((3, 4))
print((A @ B).shape)

try:
    A @ A
except ValueError:
    print("A (2 x 3) cannot multiply A (2 x 3): the inner sizes 3 and 2 do not match")

Checking shapes is the most useful debugging skill in machine learning. Most "it does not run" errors in neural network code are shape mismatches.

Order matters

With numbers, 3 × 5 = 5 × 3. With matrices, A × B is usually not equal to B × A. Often one of them is not even allowed.

Try it yourselfPython
import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[0, 1], [1, 0]])

print(A @ B)
print(B @ A)

Here B swaps things. A @ B swaps the columns of A, while B @ A swaps the rows. Different order, different result.

A neural network layer

A layer with 3 inputs and 4 neurons stores its weights in a 4 × 3 matrix W, one row per neuron. Feeding in an input vector x of size 3:

Try it yourselfPython
import numpy as np

W = np.array([
    [ 0.2, -0.5,  1.0],
    [ 1.5,  0.0, -0.3],
    [-1.5,  0.2,  0.1],
    [ 0.0,  0.3,  0.6],
])
b = np.array([0.1, 0.0, -0.2, 0.5])
x = np.array([1.0, 2.0, 3.0])

z = W @ x + b                  # 4 neurons, 4 weighted sums
output = np.maximum(0, z)      # ReLU activation
print(np.round(z, 2))
print(np.round(output, 2))

The third neuron's weighted sum is negative, so ReLU turns it into 0: that neuron stays "off" for this input. Every neuron's dot product is computed in one W @ x. A deep network is a chain of these: W₃ @ relu(W₂ @ relu(W₁ @ x)).

Common mistakes

Practice exercise

Grade a whole class with a single matrix multiplication.

Your solutionPython
import numpy as np

# 4 students (rows), 3 assessments (columns): [homework, project, exam]
X = np.array([
    [80, 70, 90],
    [60, 90, 75],
    [95, 85, 80],
    [50, 60, 55],
])
# The final grade weights each assessment: 20% homework, 30% project, 50% exam
w = np.array([0.2, 0.3, 0.5])

# 1. Work out every student's final grade with ONE matrix multiplication.
# 2. Print the shape of X, the shape of w and the shape of the result.
# 3. Print the grades.

# Output must be exactly:
# (4, 3) (3,) (4,)
# [82.  76.5 84.5 55.5]
Knowledge check

A has shape (5 × 3) and B has shape (3 × 2). What is the shape of A @ B?

Summary

  • Matrix-vector product: each row of the matrix is dotted with the vector.
  • Matrix multiplication: entry cᵢⱼ is row i of A dotted with column j of B.
  • Shape rule: (m × n) @ (n × p) gives (m × p). The inner sizes must match.
  • Order matters: A @ B is usually different from B @ A.
  • X @ w predicts for a whole dataset, and W @ x + b is a neural network layer.

Next: the identity matrix, and the inverse that undoes a matrix.

Finished reading? Tick it off to track your progress.