Start Learning Contact Us
Data Science with Python

Word Embeddings

Natural Language Processing · Lesson 24 of 30

In a bag of words, every word is its own column, so “good” and “great” are as unrelated as “good” and “bicycle”. An embedding fixes this by representing each word as a short list of numbers, a vector, placed so that words with similar meanings are near each other.

The idea with toy vectors

Real embeddings have hundreds of dimensions learned from data. These hand-made three-dimensional ones show how they are used.

import numpy as np

vectors = {
    "king":   np.array([0.9, 0.8, 0.1]),
    "queen":  np.array([0.9, 0.1, 0.8]),
    "man":    np.array([0.1, 0.9, 0.1]),
    "woman":  np.array([0.1, 0.1, 0.9]),
    "apple":  np.array([0.0, 0.5, 0.5]),
}

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print("king, queen:", round(cosine(vectors["king"], vectors["queen"]), 2))
print("king, apple:", round(cosine(vectors["king"], vectors["apple"]), 2))

Arithmetic on meaning

A famous property of learned embeddings is that directions in the space correspond to relationships. Taking the vector for “king”, subtracting “man” and adding “woman” lands near “queen”.

import numpy as np

vectors = {
    "king":   np.array([0.9, 0.8, 0.1]),
    "queen":  np.array([0.9, 0.1, 0.8]),
    "man":    np.array([0.1, 0.9, 0.1]),
    "woman":  np.array([0.1, 0.1, 0.9]),
    "apple":  np.array([0.0, 0.5, 0.5]),
}
cosine = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

target = vectors["king"] - vectors["man"] + vectors["woman"]
for word in sorted(vectors, key=lambda w: -cosine(target, vectors[w])):
    print(word, round(cosine(target, vectors[word]), 2))

The toy vectors were built to behave this way. In real embeddings the same pattern emerges from training on text, which is what made the result striking.

How embeddings are learned

The underlying idea is that a word is characterised by the words that appear around it.

word2vec, published by researchers at Google in 2013, trains a small neural network on a simple task: given a word, predict its neighbours, or given the neighbours, predict the word. The network’s internal weights for each word become that word’s vector. GloVe and fastText are related methods. fastText builds vectors from pieces of words, so it can handle words it has never seen.

Using pretrained embeddings

Training embeddings needs a very large amount of text, so most people download vectors that have already been trained.

# pip install gensim
import gensim.downloader as api

model = api.load("glove-wiki-gigaword-50")   # downloads on first use
print(model.most_similar("laptop", topn=5))
print(model.most_similar(positive=["king", "woman"], negative=["man"], topn=1))

From words to sentences

A simple way to represent a sentence is to average its word vectors.

import numpy as np

vectors = {
    "great":    np.array([0.9, 0.1]),
    "good":     np.array([0.8, 0.2]),
    "terrible": np.array([-0.9, 0.1]),
    "battery":  np.array([0.0, 0.9]),
    "screen":   np.array([0.1, 0.8]),
}

def sentence_vector(text):
    return np.mean([vectors[w] for w in text.split() if w in vectors], axis=0)

cosine = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
a, b, c = map(sentence_vector, ["great battery", "good screen", "terrible battery"])
print("great battery / good screen:      ", round(cosine(a, b), 2))
print("great battery / terrible battery: ", round(cosine(a, c), 2))

The two positive sentences are close even though they share no words, which a bag of words could not show.

One vector per word is a limit

A classic word embedding gives “bank” a single vector, whether the sentence is about a river or about money. Contextual embeddings, produced by transformer models, give a word a different vector in each sentence. They are the subject of the next lesson.

Bias

Embeddings learn from human text and absorb its biases, including stereotypes about gender and ethnicity. Check for this before using embeddings in decisions that affect people.

Where embeddings are used

  • Semantic search: finding text with similar meaning, not only matching words.
  • Recommendations: items represented as vectors.
  • Clustering documents by topic.
  • Retrieval-augmented generation, which you will build in the generative AI course.

Practice

Add a vector for “bad” to the last example and check that “bad screen” is closer to “terrible battery” than to “great battery”.