Start Learning Contact Us
Data Science with Python

Transformers for NLP

Natural Language Processing · Lesson 25 of 30

The transformer, introduced in the 2017 paper “Attention Is All You Need”, is the architecture behind today’s language models. This lesson gives you the working idea and shows how to use one.

What came before

Earlier models read text one word at a time and carried a summary forward. They struggled to connect words that were far apart, and they could not process a sentence in parallel.

Attention

In the sentence “The trophy did not fit in the suitcase because it was too big”, what does “it” refer to? To answer, you look back at the other words. Self-attention lets a model do the same: when it builds the representation of one word, it weighs every other word in the text by how relevant it is.

The calculation has three steps:

  1. Compare the current word’s vector with every word’s vector to get a score.
  2. Turn the scores into weights that sum to 1, using the softmax function.
  3. Take the weighted average of the word vectors.
import numpy as np

words = ["the", "cat", "sat"]
vectors = np.array([
    [1.0, 0.0, 0.0, 0.2],   # the
    [0.1, 1.0, 0.3, 0.0],   # cat
    [0.0, 0.6, 1.0, 0.1],   # sat
])

def softmax(x):
    e = np.exp(x - x.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

scores = vectors @ vectors.T / np.sqrt(vectors.shape[1])
weights = softmax(scores)
contextual = weights @ vectors

for word, row in zip(words, weights):
    print(f"{word:4s} attends to:", dict(zip(words, row.round(2))))
print("new vector for 'sat':", contextual[2].round(2))

Each word’s new vector is a blend of all the words, so it now carries context. A real transformer learns separate transformations of the vectors before comparing them, runs several attention calculations side by side (called heads) and stacks many layers.

Other parts of the architecture

  • Positional information. Attention alone ignores word order, so the position of each token is added to its vector.
  • Feed-forward layers. After attention, each position passes through a small neural network.
  • Stacking. The same block is repeated many times, each layer building on the last.

Tokens

Transformers do not work on whole words. A tokeniser splits text into subword tokens, so that a rare word such as “unhappiness” might become “un”, “happi” and “ness”. This keeps the vocabulary to a manageable size while still covering any text.

Three families

Family Reads Good at Example
Encoder The whole text at once Understanding: classification, search BERT
Decoder Left to right Generating text GPT models
Encoder-decoder Input in full, output left to right Translation, summarisation T5

Pretraining and fine-tuning

A transformer is first pretrained on a very large amount of text with a self-supervised task, such as predicting a hidden word or the next word. This gives it general knowledge of language. It can then be fine-tuned on a small labelled dataset for a specific job. Reusing a pretrained model in this way is called transfer learning.

Using a pretrained model

The Hugging Face Transformers library gives access to thousands of pretrained models. Run this on your own computer.

# pip install transformers torch
from transformers import pipeline

classifier = pipeline("sentiment-analysis")
print(classifier(["The screen is not great.", "Really happy with the fast delivery."]))

summarizer = pipeline("summarization")
ner = pipeline("ner", aggregation_strategy="simple")
print(ner("Asha Rao joined AshpaTech in Pune in 2025."))

Unlike the bag-of-words model in the previous lessons, a transformer classifier understands that “not great” is negative, because attention relates “not” to “great”.

Getting sentence embeddings

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(["How do I reset my password?", "I forgot my login details"])
print(embeddings.shape)
print(model.similarity(embeddings, embeddings))

These two sentences share almost no words, yet their embeddings are close.

Costs

Transformers are large. They need more memory and computing power than the classic methods, usually a graphics processor for training, and they are harder to explain. Start with TF-IDF and a linear model as a baseline, and move to a transformer when the gain justifies the cost.

What comes next

The deep learning module that follows explains how neural networks are trained. The generative AI course then returns to transformers and shows how large language models are built and used.

Practice

In the attention example, change the vector for “the” to be similar to the vector for “cat”. How do the attention weights change?