Lesson 13 of 16 · Decompositions
Eigenvalues and Eigenvectors
Understand eigenvectors as the directions a matrix only stretches, eigenvalues as the stretch factors, Av = λv, what repeated multiplication reveals, and why they matter for PCA and PageRank.
- 8 min read
- Intermediate
- Reviewed
- What eigenvectors and eigenvalues are, as a picture
- How to check the equation Av = λv
- How to find them with NumPy
- What happens when you apply a matrix again and again
- Why symmetric matrices and their eigenvectors are so important in AI
The directions that do not turn
When a matrix transforms space, almost every arrow gets knocked off its line: it turns to point somewhere new. But a few special directions stay on their own line. They only get longer, shorter or flipped.
The whole idea fits in one equation:
A v = λ v"Applying the matrix A to v gives the same result as just multiplying v by the number λ."
See them
The dashed pink lines show the eigenvector directions of the matrix. Every arrow on those lines stays on them. Move the sliders and watch the lines move:
Checking Av = λv
For A = [[2, 1], [1, 2]], try v = [1, 1] and w = [1, -1]:
import numpy as np
A = np.array([[2, 1],
[1, 2]])
v = np.array([1, 1])
w = np.array([1, -1])
u = np.array([1, 0])
print("A @ v =", A @ v, " 3 * v =", 3 * v) # same: eigenvalue 3
print("A @ w =", A @ w, " 1 * w =", 1 * w) # same: eigenvalue 1
print("A @ u =", A @ u, " not a multiple of u: u is not an eigenvector")[1, 0] turns into [2, 1], which points in a new direction, so it is not an eigenvector.
Finding them with NumPy
import numpy as np
A = np.array([[2.0, 1.0],
[1.0, 2.0]])
values, vectors = np.linalg.eigh(A) # eigh: for symmetric matrices
print("eigenvalues:", values)
for i in range(len(values)):
v = vectors[:, i] # eigenvectors are the COLUMNS
print(f"λ = {values[i]:.0f}, direction {np.round(v / v[0], 3)},",
"A v = λ v?", np.allclose(A @ v, values[i] * v))Two details worth knowing:
- The eigenvectors come back as the columns of the matrix, each scaled to length 1.
- Any multiple of an eigenvector is also an eigenvector, including its negative. Different computers may return
[0.707, 0.707]or[-0.707, -0.707]. Both are correct. Dividing by the first component, as above, shows the direction clearly.
For matrices that are not symmetric, use np.linalg.eig instead of eigh.
Applying a matrix again and again
Eigenvectors explain what happens when you apply the same matrix many times. Start with any vector, apply A repeatedly, and it swings round towards the eigenvector with the largest eigenvalue, because that direction grows fastest.
import numpy as np
A = np.array([[2.0, 1.0],
[1.0, 2.0]])
x = np.array([1.0, 0.0])
for step in range(1, 7):
x = A @ x
x = x / np.linalg.norm(x) # keep the length at 1, so we only watch the direction
print(step, np.round(x, 4))The vector settles on [0.7071, 0.7071], the eigenvector with eigenvalue 3. This simple method is called power iteration.
When there are no real eigenvectors
A rotation turns every direction, so no real arrow stays on its own line. [[0, -1], [1, 0]] (a 90° turn) has no real eigenvectors. Its eigenvalues are complex numbers, which you can safely ignore for now.
Symmetric matrices are special
This is why symmetric matrices, such as XᵀX and covariance matrices, are so friendly. Their eigenvectors form a set of perpendicular axes along which the data simply stretches, and that is exactly what PCA uses.
Common mistakes
Write a test that decides whether a vector is an eigenvector, and finds its eigenvalue.
import numpy as np
A = np.array([[4.0, 1.0],
[2.0, 3.0]])
# Write is_eigenvector(A, v). It should:
# - compute w = A @ v
# - compute the candidate eigenvalue lam = (w @ v) / (v @ v)
# - return lam if np.allclose(w, lam * v), otherwise None
# Then test the three vectors below.
candidates = [np.array([1.0, 1.0]), np.array([1.0, -2.0]), np.array([1.0, 0.0])]
# Output must be exactly:
# [1. 1.]: eigenvector, eigenvalue 5.0
# [ 1. -2.]: eigenvector, eigenvalue 2.0
# [1. 0.]: not an eigenvector
A v = 4 v for a non-zero vector v. What does this tell you?
Summary
- An eigenvector keeps its direction under a matrix. Its eigenvalue λ is the stretch factor:
Av = λv. - For 2 × 2 matrices, the eigenvalues add up to the trace (a + d) and multiply to the determinant.
- NumPy:
np.linalg.eighfor symmetric matrices,np.linalg.eigotherwise. Eigenvectors are the columns. - Repeated multiplication pulls vectors towards the dominant eigenvector (power iteration, PageRank).
- Symmetric matrices have real eigenvalues and perpendicular eigenvectors, the foundation of PCA.
Next: the singular value decomposition, which works for every matrix.
Finished reading? Tick it off to track your progress.