Velamuvi AIAcademy

Lesson 12 of 16 · Transformations

Projections

Learn to project one vector onto another, split a vector into parallel and perpendicular parts, and see why projection is the geometry behind least squares and dimensionality reduction.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What projecting a vector onto another means
  • The projection formula, built from dot products
  • How to split a vector into a parallel and a perpendicular part
  • Why least squares regression is a projection
  • How projection leads to dimensionality reduction

The shadow of a vector

Imagine the sun directly above a sloping path, and a stick leaning across it. The shadow the stick casts on the path is its projection: the part of the stick that goes along the path.

baprojection
a is projected onto the line through b. The dashed line from a to its shadow meets b's line at a right angle.

The dashed line is the key: it meets b's line at exactly 90°. That is what makes the shadow the closest point.

The formula

The projection uses dot products:

            a · b
proj_b(a) = ----- × b
            b · b

The fraction (a · b) / (b · b) says "how many b's long" the shadow is. Multiplying by b turns that number into a vector pointing along b.

Try it yourselfPython
import numpy as np

a = np.array([2.0, 3.0])
b = np.array([4.0, 1.0])

scale = (a @ b) / (b @ b)
projection = scale * b

print("how many b's:", round(scale, 4))
print("projection:  ", np.round(projection, 3))

Splitting a vector in two

Every vector can be split into a part along b and a part at right angles to b:

Try it yourselfPython
import numpy as np

a = np.array([2.0, 3.0])
b = np.array([4.0, 1.0])

along = (a @ b) / (b @ b) * b
across = a - along

print("along b:    ", np.round(along, 3))
print("across b:   ", np.round(across, 3))
print("add back up:", np.round(along + across, 3))
print("across · b: ", round(across @ b, 10) + 0.0)     # 0 means perpendicular

The perpendicular part has a dot product of 0 with b, confirming the right angle.

Least squares is a projection

Remember the Linear Systems lesson: with more equations than unknowns, there is usually no exact solution to Ax = b. Least squares finds the best Ax instead. Here is the geometry behind it:

  • Every possible Ax is a combination of the columns of A, so all of them lie in a flat region (the span of the columns).
  • b usually sticks out of that region.
  • The closest Ax to b is the projection of b onto the span of the columns.
  • The leftover error, b - Ax, is perpendicular to every column.
Try it yourselfPython
import numpy as np

# Fit a line through 4 points: each row is [x, 1] for y = m*x + c
A = np.array([[1.0, 1.0],
              [2.0, 1.0],
              [3.0, 1.0],
              [4.0, 1.0]])
b = np.array([2.0, 3.0, 5.0, 6.0])

x = np.linalg.lstsq(A, b, rcond=None)[0]     # [0] keeps just the solution
fitted = A @ x                 # the projection of b
error = b - fitted

print("m and c:", np.round(x, 3))
print("fitted: ", np.round(fitted, 3))
print("error · each column:", np.round(A.T @ error, 10) + 0.0)

The error is perpendicular to both columns: their dot products are 0. Writing that condition as Aᵀ(b - Ax) = 0 gives the normal equations AᵀAx = Aᵀb, the classic formula for linear regression.

Common mistakes

Practice exercise

Split the wind into the part along a road and the part across it.

Your solutionPython
import numpy as np

wind = np.array([6.0, 2.0])        # wind velocity
road = np.array([1.0, 0.0])        # direction of a road (east)

# 1. Project the wind onto the road direction: how much wind blows ALONG the road?
# 2. Find the part of the wind that blows ACROSS the road.
# 3. Check that the across part is perpendicular to the road (dot product 0).

# Output must be exactly:
# Along the road: [6. 0.]
# Across the road: [0. 2.]
# Perpendicular: True
Knowledge check

When a is projected onto b, what angle does the error a - proj_b(a) make with b?

Summary

  • The projection of a onto b is its shadow on b's line: (a · b)/(b · b) × b.
  • The scalar projection says how far along b the shadow reaches.
  • Any vector splits into a part along b and an orthogonal component across it.
  • Least squares projects b onto the span of A's columns, leaving an error perpendicular to them.
  • Projecting onto a few good directions is the idea behind dimensionality reduction.

You have finished the Transformations module. Next: eigenvectors, the special directions of a matrix.

Finished reading? Tick it off to track your progress.