Lesson 6 of 16 · Matrices
Matrix Multiplication
Learn matrix-vector and matrix-matrix multiplication as rows dotted with columns, the shape rule, why order matters, and how one matrix multiplication makes predictions for a whole dataset.
- 10 min read
- Intermediate
- Reviewed
- How to multiply a matrix by a vector
- How to multiply two matrices, entry by entry
- The shape rule that says when multiplication is possible
- Why A × B is usually different from B × A
- How one multiplication predicts for a whole dataset, or runs a neural network layer
Matrix times vector
You already know the dot product. Matrix multiplication is just many dot products at once.
[ 2 1 ] [ 3 ] [ 2×3 + 1×4 ] [ 10 ]
[ 0 5 ] × [ 4 ] = [ 0×3 + 5×4 ] = [ 20 ]
[ 1 1 ] [ 1×3 + 1×4 ] [ 7 ]import numpy as np
A = np.array([
[2, 1],
[0, 5],
[1, 1],
])
x = np.array([3, 4])
print(A @ x)In NumPy, @ is the matrix multiplication operator.
Predictions for a whole dataset
Here is why this matters. Put every example in the rows of a data matrix X, and the model's weights in a vector w. Then X @ w takes the dot product of every example with the weights: all the predictions in one step.
import numpy as np
# Each row is a house: [size m², bedrooms, age]
X = np.array([
[120, 3, 15],
[ 80, 2, 30],
[200, 5, 2],
])
w = np.array([40, 150, -20]) # weights from a trained model (thousands per unit)
b = 500 # bias
predictions = X @ w + b
print(predictions)Three houses, three predictions, no loop. With a million houses, the code is exactly the same.
Matrix times matrix
import numpy as np A = np.array([[1, 2], [3, 4]]) B = np.array([[5, 6], [7, 8]]) print(A @ B)
The shape rule
You can only multiply when the shapes fit together.
(m × n) @ (n × p) → (m × p)
└─same─┘| A shape | B shape | Allowed? | Result shape |
|---|---|---|---|
| 2 × 3 | 3 × 4 | yes | 2 × 4 |
| 100 × 20 | 20 × 1 | yes | 100 × 1 |
| 2 × 3 | 2 × 3 | no: 3 ≠ 2 |
import numpy as np
A = np.ones((2, 3))
B = np.ones((3, 4))
print((A @ B).shape)
try:
A @ A
except ValueError:
print("A (2 x 3) cannot multiply A (2 x 3): the inner sizes 3 and 2 do not match")Checking shapes is the most useful debugging skill in machine learning. Most "it does not run" errors in neural network code are shape mismatches.
Order matters
With numbers, 3 × 5 = 5 × 3. With matrices, A × B is usually not equal to B × A. Often one of them is not even allowed.
import numpy as np A = np.array([[1, 2], [3, 4]]) B = np.array([[0, 1], [1, 0]]) print(A @ B) print(B @ A)
Here B swaps things. A @ B swaps the columns of A, while B @ A swaps the rows. Different order, different result.
A neural network layer
A layer with 3 inputs and 4 neurons stores its weights in a 4 × 3 matrix W, one row per neuron. Feeding in an input vector x of size 3:
import numpy as np
W = np.array([
[ 0.2, -0.5, 1.0],
[ 1.5, 0.0, -0.3],
[-1.5, 0.2, 0.1],
[ 0.0, 0.3, 0.6],
])
b = np.array([0.1, 0.0, -0.2, 0.5])
x = np.array([1.0, 2.0, 3.0])
z = W @ x + b # 4 neurons, 4 weighted sums
output = np.maximum(0, z) # ReLU activation
print(np.round(z, 2))
print(np.round(output, 2))The third neuron's weighted sum is negative, so ReLU turns it into 0: that neuron stays "off" for this input. Every neuron's dot product is computed in one W @ x. A deep network is a chain of these: W₃ @ relu(W₂ @ relu(W₁ @ x)).
Common mistakes
Grade a whole class with a single matrix multiplication.
import numpy as np
# 4 students (rows), 3 assessments (columns): [homework, project, exam]
X = np.array([
[80, 70, 90],
[60, 90, 75],
[95, 85, 80],
[50, 60, 55],
])
# The final grade weights each assessment: 20% homework, 30% project, 50% exam
w = np.array([0.2, 0.3, 0.5])
# 1. Work out every student's final grade with ONE matrix multiplication.
# 2. Print the shape of X, the shape of w and the shape of the result.
# 3. Print the grades.
# Output must be exactly:
# (4, 3) (3,) (4,)
# [82. 76.5 84.5 55.5]
A has shape (5 × 3) and B has shape (3 × 2). What is the shape of A @ B?
Summary
- Matrix-vector product: each row of the matrix is dotted with the vector.
- Matrix multiplication: entry cᵢⱼ is row i of A dotted with column j of B.
- Shape rule: (m × n) @ (n × p) gives (m × p). The inner sizes must match.
- Order matters:
A @ Bis usually different fromB @ A. X @ wpredicts for a whole dataset, andW @ x + bis a neural network layer.
Next: the identity matrix, and the inverse that undoes a matrix.
Finished reading? Tick it off to track your progress.