Velamuvi AIAcademy

Lesson 16 of 16 · Decompositions

Linear Algebra in NumPy

A practical NumPy guide for linear algebra: creating arrays, reshaping, broadcasting, vectorised code and the np.linalg toolbox, ending with a mini semantic search engine built from scratch.

  • 9 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • How to create and reshape the arrays you need
  • How broadcasting combines arrays of different shapes
  • Why vectorised code beats Python loops
  • The np.linalg functions you will use most
  • How to build a small semantic search engine with matrices

Creating arrays

Try it yourselfPython
import numpy as np

print(np.zeros((2, 3)))            # all zeros, shape 2 x 3
print(np.ones(3))                   # a vector of ones
print(np.eye(2))                    # the identity matrix
print(np.arange(6))                 # 0 to 5
print(np.linspace(0, 1, 5))         # 5 evenly spaced numbers from 0 to 1

Reshaping

Try it yourselfPython
import numpy as np

a = np.arange(6)
print(a.reshape(2, 3))
print(a.reshape(3, -1))             # -1: "work out this size for me"

column = a.reshape(-1, 1)           # turn a 1-D array into a column vector
print(column.shape)

Reshaping is constant in machine learning: flattening a 28 × 28 image into a vector of 784, or arranging a long list of numbers into a batch.

Broadcasting

Try it yourselfPython
import numpy as np

X = np.array([[1.0, 200.0],
              [2.0, 300.0],
              [3.0, 400.0]])          # shape (3, 2)

means = X.mean(axis=0)               # shape (2,)
stds = X.std(axis=0)

X_scaled = (X - means) / stds        # (3, 2) minus (2,): the means are used for every row
print(np.round(X_scaled, 3) + 0.0)

Broadcasting is how you centre and scale every column of a dataset in one line, with no loop. This standardisation is the usual first step before PCA, k-nearest neighbours or training a neural network.

Vectorise: avoid Python loops

Both versions below compute the same predictions. The vectorised one is one line and, on large data, often hundreds of times faster:

Try it yourselfPython
import numpy as np

X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
w = np.array([0.5, -1.0])

# Loop version
predictions = []
for row in X:
    total = 0.0
    for j in range(len(w)):
        total += row[j] * w[j]
    predictions.append(float(total))     # float() keeps the printout simple
print(predictions)

# Vectorised version
print(X @ w)

The np.linalg toolbox

FunctionWhat it doesLesson
np.linalg.norm(v)length of a vectorVector Length
A @ Bmatrix multiplicationMatrix Multiplication
np.linalg.inv(A)inverseIdentity and Inverse
np.linalg.solve(A, b)solve Ax = bLinear Systems
np.linalg.lstsq(A, b)least squaresLinear Systems
np.linalg.det(A)determinantRank and Determinant
np.linalg.matrix_rank(A)rankRank and Determinant
np.linalg.eigh(A)eigenvalues and vectors (symmetric)Eigenvalues
np.linalg.svd(A)singular value decompositionSVD

Mini project: a semantic search engine

Time to use the whole course at once. Each document below has a small "embedding" vector, as a real embedding model would produce (real ones have hundreds of dimensions). The search engine:

  1. stores all document vectors in one matrix,
  2. normalises every row to length 1 (broadcasting),
  3. scores every document against the query with one matrix-vector product (cosine similarity),
  4. sorts the scores to rank the results.
Try it yourselfPython
import numpy as np

titles = ["How to bake bread", "Training a neural network", "Best pasta recipes",
          "Linear algebra for ML", "Making pizza dough"]

# Columns: [cooking, machine learning, maths]
E = np.array([
    [0.90, 0.05, 0.05],
    [0.05, 0.90, 0.40],
    [0.95, 0.00, 0.05],
    [0.00, 0.60, 0.80],
    [0.85, 0.05, 0.10],
])

E_unit = E / np.linalg.norm(E, axis=1, keepdims=True)    # every row now has length 1


def search(query_vector, top=3):
    q = query_vector / np.linalg.norm(query_vector)
    scores = E_unit @ q                     # cosine similarity with every document at once
    order = np.argsort(scores)[::-1]        # indices from best to worst
    for i in order[:top]:
        print(f"  {scores[i]:.3f}  {titles[i]}")


print("Query: 'maths behind deep learning'")
search(np.array([0.0, 0.7, 0.7]))
print("Query: 'something to cook tonight'")
search(np.array([1.0, 0.0, 0.0]))

That is, in miniature, how vector databases answer millions of questions a day. The only differences in a real system are longer vectors from an embedding model and clever indexes that avoid checking every document.

Common mistakes

Practice exercise

Build a "similar products" feature from a similarity matrix.

Your solutionPython
import numpy as np

# Embeddings of 4 products: [sporty, formal, casual]
names = ["running shoes", "office shirt", "hoodie", "suit"]
E = np.array([
    [0.9, 0.0, 0.4],
    [0.0, 0.9, 0.2],
    [0.3, 0.0, 0.9],
    [0.0, 1.0, 0.0],
])

# 1. Normalise every row of E to length 1 (use keepdims=True).
# 2. Compute the 4 x 4 cosine similarity matrix S = E_unit @ E_unit.T, rounded to 2 places.
# 3. For "office shirt" (row 1), print the most similar OTHER product.

# Output must be exactly:
# [[1.   0.09 0.67 0.  ]
#  [0.09 1.   0.21 0.98]
#  [0.67 0.21 1.   0.  ]
#  [0.   0.98 0.   1.  ]]
# Most like office shirt: suit
Knowledge check

X has shape (100, 4) and m has shape (4,). What shape is X - m?

Summary

  • Create arrays with zeros, ones, eye, arange and linspace, and change shape with reshape.
  • Broadcasting combines different shapes, such as subtracting column means from a whole dataset.
  • Vectorised code replaces Python loops with fast whole-array operations.
  • np.linalg covers norms, solving, inverses, determinants, rank, eigenvectors and the SVD.
  • A semantic search engine is just normalising, one matrix-vector product and a sort.

You have finished Linear Algebra

You can now think of data as vectors and matrices, compute with dot products and matrix multiplication, see matrices as transformations of space, solve linear systems and least-squares problems, and understand the determinant, rank, projections, eigenvectors, the SVD and PCA. These are the foundations of every machine learning model, from linear regression to large language models.

Where next?

Finished reading? Tick it off to track your progress.