Start Learning Contact Us
Data Science with Python

Sequence Models and the Road to Transformers

Deep Learning · Lesson 30 of 30

Text, speech, sensor readings and stock prices are sequences: the order of the items carries meaning. This lesson follows how deep learning learned to handle them, which leads directly to the models behind generative AI.

Recurrent neural networks

A recurrent neural network (RNN) reads a sequence one item at a time. It keeps a hidden state, a vector that summarises what it has seen so far, and updates it at each step.

import numpy as np

rng = np.random.default_rng(0)
W_input = rng.normal(0, 0.5, (3, 4))    # input vector (3) to hidden state (4)
W_hidden = rng.normal(0, 0.5, (4, 4))   # previous hidden state to new hidden state

sequence = rng.normal(0, 1, (5, 3))     # 5 steps, each a vector of 3 numbers
hidden = np.zeros(4)

for step, x in enumerate(sequence, 1):
    hidden = np.tanh(x @ W_input + hidden @ W_hidden)
    print(f"step {step}: hidden state {hidden.round(2)}")

The same weights are used at every step, so the network can handle sequences of any length. After the last step, the hidden state represents the whole sequence.

The problem with long sequences

Training an RNN sends gradients backwards through every step. Over many steps they tend to shrink towards zero, which is called the vanishing gradient problem. The network then fails to connect a word with something that appeared much earlier.

LSTM and GRU

The long short-term memory network (LSTM) adds a separate memory and gates, small networks that decide what to store, what to forget and what to output. The GRU is a simpler variant. Both remember over much longer spans than a plain RNN, and for years they were the standard choice for language, speech and time series.

import torch
from torch import nn

lstm = nn.LSTM(input_size=3, hidden_size=16, batch_first=True)
batch = torch.randn(8, 20, 3)             # 8 sequences, 20 steps, 3 features
outputs, (hidden, cell) = lstm(batch)
print(outputs.shape)                      # torch.Size([8, 20, 16])

Two remaining limits

  • Speed. Step 20 cannot be calculated until step 19 is done, so an RNN cannot use the parallel power of a GPU well.
  • A bottleneck. Squeezing a long document into one hidden state loses information.

Attention

Attention was first added to RNN translation models. Instead of relying on a single summary vector, the model could look back at every input word and focus on the relevant ones while producing each output word.

In 2017 the paper “Attention Is All You Need” showed that the recurrence could be removed entirely. The resulting architecture, the transformer, relies on attention alone. You met its mechanism in the NLP module.

RNN or LSTM Transformer
Processes a sequence One step after another All positions at once
Connects distant items With difficulty Directly, through attention
Training on GPUs Slow Fast, highly parallel
Needs position information added No Yes

Because transformers train in parallel, they can be trained on far more data than earlier models, which is what made very large models practical.

Foundation models

A foundation model is a large model pretrained on broad data that can be adapted to many tasks. Large language models are the best-known kind. The pattern you saw with CNNs applies here on a much larger scale:

  1. Pretrain once on a huge dataset, at great cost.
  2. Adapt cheaply: fine-tune on a small dataset, or simply describe the task in a prompt.

Most practitioners now work at step 2. Knowing what happens in step 1, which this module covered, lets you reason about what these models can and cannot do.

Where sequence models are still used

Transformers dominate language. For time series such as sales or sensor readings, simpler approaches are often as good or better: classical statistical forecasting, gradient-boosted trees on lagged features, and sometimes LSTMs. Try a simple method first.

Training deep networks in practice

  • Hardware. Deep learning relies on GPUs. Free hosted notebooks provide enough for learning.
  • Normalisation layers and residual connections, which let a layer’s input skip past it, are what make very deep networks trainable.
  • Learning-rate schedules start with larger steps and reduce them over time.
  • Mixed precision uses smaller number formats to save memory and time.
  • Experiment tracking. Record the settings and results of every run. You will not remember them.

Responsible use

Deep models learn whatever patterns are in their data, including errors and unfair bias. They can be confidently wrong, and it is hard to explain an individual prediction. Test on data that reflects real use, measure performance separately for different groups of people, and keep a person in the loop for decisions that matter.

What you have learned in this course

  • Python basics: the language, NumPy and pandas.
  • Statistics: describing data and reasoning about uncertainty.
  • Machine learning: training, evaluating and tuning models.
  • NLP: turning text into numbers and classifying it.
  • Deep learning: neural networks, how they are trained and the main architectures.

What comes next

The remaining three courses build on the transformer. Prompt Engineering teaches you to get reliable results from large language models. Generative AI and LLMs explains how they work and how to build applications with them. Agentic AI shows how to give them tools and let them carry out multi-step work.

Practice

In the RNN example, feed the same sequence in reverse order. Is the final hidden state the same? What does that tell you about whether the network uses order?