Velamuvi AIAcademy

Lesson 10 of 16 · Transformations

Linear Systems

Learn to write systems of linear equations as Ax = b, see their solutions as crossing lines, recognise when there is one, none or infinitely many solutions, and solve them with NumPy, including least squares.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • What a system of linear equations is
  • How to write one as Ax = b
  • What one, no or infinitely many solutions look like
  • How to solve systems with np.linalg.solve
  • How least squares finds the best answer when there is no exact one

Several equations at once

A café sells tea and coffee. One morning, 3 teas and 2 coffees cost 340. Another morning, 1 tea and 4 coffees cost 380. What does each drink cost?

3t + 2c = 340
1t + 4c = 380

Writing it as Ax = b

[ 3  2 ]   [ t ]     [ 340 ]
[ 1  4 ] × [ c ]  =  [ 380 ]
    A        x          b

Each row of A, dotted with x, gives one equation. In this form, a computer can solve thousands of equations as easily as two.

Try it yourselfPython
import numpy as np

A = np.array([[3, 2],
              [1, 4]])
b = np.array([340, 380])

x = np.linalg.solve(A, b)
print("tea =", round(x[0], 6), " coffee =", round(x[1], 6))
print("check:", np.round(A @ x, 6))

Tea costs 60 and coffee 80. Checking by putting the answer back in is always a good habit.

A picture: crossing lines

Each equation with two unknowns is a straight line. The solution is the point where the lines cross, because only that point lies on both.

There are only three possibilities:

LinesNumber of solutionsMatrix A
cross at one pointexactly oneinvertible (determinant not 0)
parallel, never meetnonesingular
the same lineinfinitely manysingular
Try it yourselfPython
import numpy as np

# Parallel lines: x + y = 2 and x + y = 5 can never both be true
A = np.array([[1, 1],
              [1, 1]])
b = np.array([2, 5])

print("determinant:", round(np.linalg.det(A), 6) + 0.0)
if np.linalg.matrix_rank(A) < A.shape[0]:
    print("No single solution: the matrix is singular")
else:
    print(np.linalg.solve(A, b))

Check the rank (or the determinant) before calling np.linalg.solve. On some setups, including this browser, a singular matrix does not raise an error: you just get nan values back.

Bigger systems

Three unknowns need three equations. The code does not change:

Try it yourselfPython
import numpy as np

# A bakery: flour, sugar and butter in grams for three recipes,
# and the totals used today. How many batches of each recipe were made?
A = np.array([[200, 100, 300],     # flour per batch of cake, cookies, bread
              [150, 200,  20],     # sugar
              [100, 150,  40]])    # butter
b = np.array([1600, 1290, 980])    # total flour, sugar, butter used

batches = np.linalg.solve(A, b)
print(np.round(batches, 6) + 0.0)

When there is no exact answer: least squares

Real data is noisy. You often have more equations than unknowns, and no single answer satisfies them all. For example, fitting a line y = m·x + c through 5 measured points gives 5 equations but only 2 unknowns.

Try it yourselfPython
import numpy as np

# Measured points that lie roughly on a line
hours = np.array([1, 2, 3, 4, 5])
scores = np.array([52, 58, 61, 70, 74])

# Each row says: m * hours + c * 1 = score
A = np.column_stack([hours, np.ones(5)])
solution, residuals, rank, singular_values = np.linalg.lstsq(A, scores, rcond=None)
m, c = solution

print(f"best line: score = {m:.1f} × hours + {c:.1f}")
print("prediction for 6 hours:", round(m * 6 + c, 1))

You just did linear regression with linear algebra. The best-fit line is the least-squares solution of an overdetermined system.

Common mistakes

Practice exercise

Turn a word problem into Ax = b and solve it.

Your solutionPython
import numpy as np

# A cinema sold adult and child tickets.
# Monday:  4 adult + 2 child tickets = 1000
# Tuesday: 3 adult + 5 child tickets = 1100
# 1. Write the system as A and b.
# 2. Solve it with np.linalg.solve.
# 3. Print each price, and check the answer with A @ prices.

# Output must be exactly:
# Adult: 200.0
# Child: 100.0
# Check: [1000. 1100.]
Knowledge check

Two equations in two unknowns describe parallel lines. How many solutions are there?

Summary

  • A system of linear equations is written Ax = b: coefficients, unknowns and results.
  • Each solution is where all the equations hold at once, such as where lines cross.
  • There is exactly one solution, none or infinitely many. One means A is invertible.
  • Solve square systems with np.linalg.solve.
  • With more equations than unknowns, least squares (np.linalg.lstsq) gives the best fit. That is linear regression.

Next: the determinant and rank, which measure how much a matrix squashes space.

Finished reading? Tick it off to track your progress.