Velamuvi AIAcademy

Lesson 2 of 16 · Vectors

Vector Addition and Scaling

Learn to add, subtract and scale vectors, see each operation as a picture, and understand linear combinations and span, the ideas behind averages, embeddings and neural networks.

  • 8 min read
  • Intermediate
  • Reviewed
In this lesson you will learn
  • How to add and subtract vectors, and what that looks like
  • How multiplying by a scalar stretches, shrinks or flips a vector
  • What a linear combination is
  • What the span of vectors means
  • How these operations appear in machine learning

Adding vectors

As a picture, addition means tip to tail: walk along the first arrow, then walk along the second from where you stopped. The sum is the arrow from the start to where you end up.

aba + b
a = [3, 1], then b = [1, 2] from the tip of a. The sum a + b = [4, 3] goes straight from start to finish.
Try it yourselfPython
import numpy as np

a = np.array([3, 1])
b = np.array([1, 2])

print(a + b)
print(b + a)        # the order does not matter

Think of directions in a city: "3 blocks east and 1 north, then 1 east and 2 north" ends in the same place as "4 east and 3 north".

Subtracting vectors

Subtraction answers "how do I get from here to there?". It is how you measure the difference between two data points, such as a prediction and the true answer.

Try it yourselfPython
import numpy as np

predicted = np.array([210, 3.1])
actual = np.array([200, 3.0])

difference = predicted - actual
print("Error vector:", difference)

Scaling a vector

2vv0.5v-v
v = [2, 1] scaled by 2, 0.5 and -1. A negative scalar flips the arrow to point the other way.
ScalarEffect
bigger than 1stretches the vector
between 0 and 1shrinks it
0gives the zero vector [0, 0]
negativeflips it to the opposite direction, then stretches or shrinks

The word scalar comes from this: a scalar scales a vector.

Linear combinations

Put scaling and adding together and you get the most important pattern in linear algebra.

Try it yourselfPython
import numpy as np

v1 = np.array([1, 0])
v2 = np.array([0, 1])

combo = 3 * v1 + 2 * v2
print(combo)

# Mixing ingredients: 2 portions of recipe A and 1 portion of recipe B
recipe_a = np.array([200, 50, 10])      # grams of flour, sugar, butter
recipe_b = np.array([100, 0, 40])
print(2 * recipe_a + 1 * recipe_b)

A neural network layer is built from linear combinations: each neuron multiplies its inputs by weights and adds them up.

Span

  • The span of [1, 0] alone is just the horizontal line: you can only stretch along it.
  • The span of [1, 0] and [0, 1] is the whole 2D plane: any [x, y] is x[1, 0] + y[0, 1].
  • The span of [1, 2] and [2, 4] is only a line, because the second is just double the first. It adds no new direction.

Common mistakes

Practice exercise

Find the centre of a group of users with a linear combination, as clustering algorithms do.

Your solutionPython
import numpy as np

# Three users described by [hours of sport, hours of music, hours of cooking] watched per week
users = [np.array([4, 1, 0]), np.array([6, 0, 2]), np.array([2, 2, 1])]

# 1. Work out the centroid (average vector) of the three users with a linear combination.
#    Add them up, then multiply by 1/3. Round to 2 places with np.round.
# 2. Work out how far the first user is from the centroid: users[0] - centroid, rounded to 2 places.

# Output must be exactly:
# Centroid: [4. 1. 1.]
# First user minus centroid: [ 0.  0. -1.]
Knowledge check

What is 3 × [2, -1] + [1, 4]?

Summary

  • Add vectors component by component. As arrows, place them tip to tail.
  • Subtract to get the arrow from one point to another, such as an error.
  • Scalar multiplication stretches, shrinks or flips a vector.
  • A linear combination is a weighted sum of vectors. Averages and neuron outputs are linear combinations.
  • The span is everything a set of vectors can reach.

Next: the dot product, the single most used operation in machine learning.

Finished reading? Tick it off to track your progress.