Lesson 12 of 16 · Transformations
Projections
Learn to project one vector onto another, split a vector into parallel and perpendicular parts, and see why projection is the geometry behind least squares and dimensionality reduction.
- 8 min read
- Intermediate
- Reviewed
- What projecting a vector onto another means
- The projection formula, built from dot products
- How to split a vector into a parallel and a perpendicular part
- Why least squares regression is a projection
- How projection leads to dimensionality reduction
The shadow of a vector
Imagine the sun directly above a sloping path, and a stick leaning across it. The shadow the stick casts on the path is its projection: the part of the stick that goes along the path.
The dashed line is the key: it meets b's line at exactly 90°. That is what makes the shadow the closest point.
The formula
The projection uses dot products:
a · b
proj_b(a) = ----- × b
b · bThe fraction (a · b) / (b · b) says "how many b's long" the shadow is. Multiplying by b turns that number into a vector pointing along b.
import numpy as np
a = np.array([2.0, 3.0])
b = np.array([4.0, 1.0])
scale = (a @ b) / (b @ b)
projection = scale * b
print("how many b's:", round(scale, 4))
print("projection: ", np.round(projection, 3))Splitting a vector in two
Every vector can be split into a part along b and a part at right angles to b:
import numpy as np
a = np.array([2.0, 3.0])
b = np.array([4.0, 1.0])
along = (a @ b) / (b @ b) * b
across = a - along
print("along b: ", np.round(along, 3))
print("across b: ", np.round(across, 3))
print("add back up:", np.round(along + across, 3))
print("across · b: ", round(across @ b, 10) + 0.0) # 0 means perpendicularThe perpendicular part has a dot product of 0 with b, confirming the right angle.
Least squares is a projection
Remember the Linear Systems lesson: with more equations than unknowns, there is usually no exact solution to Ax = b. Least squares finds the best Ax instead. Here is the geometry behind it:
- Every possible Ax is a combination of the columns of A, so all of them lie in a flat region (the span of the columns).
- b usually sticks out of that region.
- The closest Ax to b is the projection of b onto the span of the columns.
- The leftover error, b - Ax, is perpendicular to every column.
import numpy as np
# Fit a line through 4 points: each row is [x, 1] for y = m*x + c
A = np.array([[1.0, 1.0],
[2.0, 1.0],
[3.0, 1.0],
[4.0, 1.0]])
b = np.array([2.0, 3.0, 5.0, 6.0])
x = np.linalg.lstsq(A, b, rcond=None)[0] # [0] keeps just the solution
fitted = A @ x # the projection of b
error = b - fitted
print("m and c:", np.round(x, 3))
print("fitted: ", np.round(fitted, 3))
print("error · each column:", np.round(A.T @ error, 10) + 0.0)The error is perpendicular to both columns: their dot products are 0. Writing that condition as Aᵀ(b - Ax) = 0 gives the normal equations AᵀAx = Aᵀb, the classic formula for linear regression.
Common mistakes
Split the wind into the part along a road and the part across it.
import numpy as np wind = np.array([6.0, 2.0]) # wind velocity road = np.array([1.0, 0.0]) # direction of a road (east) # 1. Project the wind onto the road direction: how much wind blows ALONG the road? # 2. Find the part of the wind that blows ACROSS the road. # 3. Check that the across part is perpendicular to the road (dot product 0). # Output must be exactly: # Along the road: [6. 0.] # Across the road: [0. 2.] # Perpendicular: True
When a is projected onto b, what angle does the error a - proj_b(a) make with b?
Summary
- The projection of a onto b is its shadow on b's line:
(a · b)/(b · b) × b. - The scalar projection says how far along b the shadow reaches.
- Any vector splits into a part along b and an orthogonal component across it.
- Least squares projects b onto the span of A's columns, leaving an error perpendicular to them.
- Projecting onto a few good directions is the idea behind dimensionality reduction.
You have finished the Transformations module. Next: eigenvectors, the special directions of a matrix.
Finished reading? Tick it off to track your progress.