Velamuvi AIAcademy

Lesson 15 of 16 · Decompositions

PCA Intuition

Understand principal component analysis without fear: centring, variance, principal components as the directions of greatest spread, explained variance, and PCA step by step in NumPy.

  • 7 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What dimensionality reduction is and why it helps
  • What variance and principal components are
  • The steps of PCA, from centring to projecting
  • How to read the explained variance ratio
  • When PCA is the right tool, and when it is not

Too many columns

Real datasets often have dozens, hundreds or thousands of features. Many of them move together: a person's height and arm span, a house's size and its number of rooms. Keeping every column wastes memory and time, and it makes the data impossible to look at.

The idea in one picture

Look at these points. They spread out mostly along one slanted direction:

PC1PC2
The data spreads mostly along PC1. Along PC2, at a right angle, it hardly varies at all.

Two numbers (x and y) describe each point. But almost all the information is in one number: how far along the PC1 direction the point lies. Keep that, drop the rest, and you have squeezed 2 features into 1 while losing very little.

Variance: measuring spread

Principal components are always perpendicular to each other, so together they form a new set of axes, rotated to fit the data.

PCA step by step

  1. Centre the data: subtract each feature's mean, so the cloud sits around the origin.
  2. Find the directions of greatest spread. These are the eigenvectors of the covariance matrix, or equivalently the right singular vectors from the SVD of the centred data.
  3. Measure how much variance each direction explains, using the eigenvalues or squared singular values.
  4. Project the data onto the top k directions to get the new, smaller set of features.
Try it yourselfPython
import numpy as np

X = np.array([[-3, -2.5], [-2, -2.2], [-1, -0.6], [0, 0.4], [1, 0.7], [2, 2.3],
              [3, 2.6], [-1.5, -1.8], [1.5, 1.1], [0.5, -0.2], [-2.5, -1.4], [2.5, 1.6]])

# 1. Centre
X_centred = X - X.mean(axis=0)

# 2. Directions of greatest spread, from the SVD
U, S, Vt = np.linalg.svd(X_centred, full_matrices=False)
pc1 = Vt[0]
if pc1[0] < 0:          # the sign of a direction is arbitrary, so make it point right
    pc1 = -pc1
print("PC1 direction:", np.round(pc1, 3))

# 3. How much variance each component explains
explained = S ** 2 / np.sum(S ** 2)
print("explained variance:", np.round(explained, 3))

# 4. Project onto PC1: one number per point instead of two
scores = X_centred @ pc1
print("new 1-D feature:", np.round(scores[:4], 2), "...")

PC1 explains about 98% of the variance. Replacing two columns with one keeps 98% of the information.

Why PCA works: the linear algebra

Everything from this course comes together here:

  • Each example is a vector, and the dataset is a matrix X.
  • Centring and computing XᵀX / (n - 1) gives the covariance matrix, which is symmetric (the Transpose lesson).
  • Its eigenvectors are perpendicular directions, and its eigenvalues are the variances along them (the Eigen lesson).
  • The SVD of the centred data gives the same directions more stably (the SVD lesson).
  • Reducing the data is a projection onto the top directions (the Projections lesson).

Common mistakes

Practice exercise

Decide how many principal components to keep for 90% of the variance.

Your solutionPython
import numpy as np

# The variance along each principal component of a dataset (largest first)
variances = np.array([6.0, 2.5, 0.9, 0.4, 0.2])

# 1. Work out the explained variance ratio of each component, rounded to 3 places.
# 2. Work out the running total of the ratios, rounded to 3 places.
# 3. Print the number of components needed to keep at least 90% of the variance.

# Output must be exactly:
# ratios: [0.6  0.25 0.09 0.04 0.02]
# running total: [0.6  0.85 0.94 0.98 1.  ]
# components for 90%: 3
Knowledge check

What is the first principal component of a dataset?

Summary

  • Dimensionality reduction keeps the important information in fewer features. PCA is the classic method.
  • Variance measures spread. Principal components are perpendicular directions of greatest variance.
  • Steps: centre, find the directions (eigenvectors of the covariance matrix, or SVD), measure the explained variance ratio, project.
  • Scale features first, and remember PCA only finds straight directions and ignores labels.

Next: a tour of the NumPy tools that turn all of this into fast, short code.

Finished reading? Tick it off to track your progress.