Velamuvi AIAcademy

Lesson 13 of 16 · Decompositions

Eigenvalues and Eigenvectors

Understand eigenvectors as the directions a matrix only stretches, eigenvalues as the stretch factors, Av = λv, what repeated multiplication reveals, and why they matter for PCA and PageRank.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What eigenvectors and eigenvalues are, as a picture
  • How to check the equation Av = λv
  • How to find them with NumPy
  • What happens when you apply a matrix again and again
  • Why symmetric matrices and their eigenvectors are so important in AI

The directions that do not turn

When a matrix transforms space, almost every arrow gets knocked off its line: it turns to point somewhere new. But a few special directions stay on their own line. They only get longer, shorter or flipped.

The whole idea fits in one equation:

A v = λ v

"Applying the matrix A to v gives the same result as just multiplying v by the number λ."

See them

The dashed pink lines show the eigenvector directions of the matrix. Every arrow on those lines stays on them. Move the sliders and watch the lines move:

îĵ
[[2, 1], [1, 2]]î lands on (2, 1), ĵ lands on (1, 2)determinant: 3eigenvalues: 3 and 1
For [[2, 1], [1, 2]], the eigenvectors lie along y = x (stretched by 3) and y = -x (stretched by 1). Try b = -1, c = 1 to see a rotation-like matrix lose its real eigenvectors.

Checking Av = λv

For A = [[2, 1], [1, 2]], try v = [1, 1] and w = [1, -1]:

Try it yourselfPython
import numpy as np

A = np.array([[2, 1],
              [1, 2]])

v = np.array([1, 1])
w = np.array([1, -1])
u = np.array([1, 0])

print("A @ v =", A @ v, "  3 * v =", 3 * v)     # same: eigenvalue 3
print("A @ w =", A @ w, " 1 * w =", 1 * w)       # same: eigenvalue 1
print("A @ u =", A @ u, "  not a multiple of u: u is not an eigenvector")

[1, 0] turns into [2, 1], which points in a new direction, so it is not an eigenvector.

Finding them with NumPy

Try it yourselfPython
import numpy as np

A = np.array([[2.0, 1.0],
              [1.0, 2.0]])

values, vectors = np.linalg.eigh(A)      # eigh: for symmetric matrices
print("eigenvalues:", values)

for i in range(len(values)):
    v = vectors[:, i]                      # eigenvectors are the COLUMNS
    print(f"λ = {values[i]:.0f}, direction {np.round(v / v[0], 3)},",
          "A v = λ v?", np.allclose(A @ v, values[i] * v))

Two details worth knowing:

  • The eigenvectors come back as the columns of the matrix, each scaled to length 1.
  • Any multiple of an eigenvector is also an eigenvector, including its negative. Different computers may return [0.707, 0.707] or [-0.707, -0.707]. Both are correct. Dividing by the first component, as above, shows the direction clearly.

For matrices that are not symmetric, use np.linalg.eig instead of eigh.

Applying a matrix again and again

Eigenvectors explain what happens when you apply the same matrix many times. Start with any vector, apply A repeatedly, and it swings round towards the eigenvector with the largest eigenvalue, because that direction grows fastest.

Try it yourselfPython
import numpy as np

A = np.array([[2.0, 1.0],
              [1.0, 2.0]])
x = np.array([1.0, 0.0])

for step in range(1, 7):
    x = A @ x
    x = x / np.linalg.norm(x)       # keep the length at 1, so we only watch the direction
    print(step, np.round(x, 4))

The vector settles on [0.7071, 0.7071], the eigenvector with eigenvalue 3. This simple method is called power iteration.

When there are no real eigenvectors

A rotation turns every direction, so no real arrow stays on its own line. [[0, -1], [1, 0]] (a 90° turn) has no real eigenvectors. Its eigenvalues are complex numbers, which you can safely ignore for now.

Symmetric matrices are special

This is why symmetric matrices, such as XᵀX and covariance matrices, are so friendly. Their eigenvectors form a set of perpendicular axes along which the data simply stretches, and that is exactly what PCA uses.

Common mistakes

Practice exercise

Write a test that decides whether a vector is an eigenvector, and finds its eigenvalue.

Your solutionPython
import numpy as np

A = np.array([[4.0, 1.0],
              [2.0, 3.0]])

# Write is_eigenvector(A, v). It should:
#   - compute w = A @ v
#   - compute the candidate eigenvalue lam = (w @ v) / (v @ v)
#   - return lam if np.allclose(w, lam * v), otherwise None
# Then test the three vectors below.
candidates = [np.array([1.0, 1.0]), np.array([1.0, -2.0]), np.array([1.0, 0.0])]

# Output must be exactly:
# [1. 1.]: eigenvector, eigenvalue 5.0
# [ 1. -2.]: eigenvector, eigenvalue 2.0
# [1. 0.]: not an eigenvector
Knowledge check

A v = 4 v for a non-zero vector v. What does this tell you?

Summary

  • An eigenvector keeps its direction under a matrix. Its eigenvalue λ is the stretch factor: Av = λv.
  • For 2 × 2 matrices, the eigenvalues add up to the trace (a + d) and multiply to the determinant.
  • NumPy: np.linalg.eigh for symmetric matrices, np.linalg.eig otherwise. Eigenvectors are the columns.
  • Repeated multiplication pulls vectors towards the dominant eigenvector (power iteration, PageRank).
  • Symmetric matrices have real eigenvalues and perpendicular eigenvectors, the foundation of PCA.

Next: the singular value decomposition, which works for every matrix.

Finished reading? Tick it off to track your progress.