Text Preprocessing

Natural language processing (NLP) is the field of getting computers to work with human language. Text is messy: the same idea can be written in countless ways. Preprocessing reduces that variety so that patterns can be found.

Normalising case and whitespace

text = "  The Battery is GREAT!!  Great battery, great price.  "

clean = " ".join(text.lower().split())
print(clean)

lower() makes “GREAT” and “great” the same word. split() followed by join collapses repeated spaces.

Tokenisation

A token is a unit of text, usually a word. Splitting on spaces leaves punctuation attached, so a regular expression does a better job.

import re

text = "The battery is great!! Great battery, great price. It's worth $49.99."

print(text.lower().split())
print(re.findall(r"[a-z0-9']+", text.lower()))

The pattern [a-z0-9']+ matches runs of letters, digits and apostrophes.

Counting words

import re
from collections import Counter

text = "The battery is great. Great battery, great price. The screen is not great."
tokens = re.findall(r"[a-z']+", text.lower())

print(Counter(tokens).most_common(5))

Stop words

Words such as “the” and “is” are frequent in all text and carry little meaning on their own. Removing these stop words often helps when you are counting words.

import re
from collections import Counter

stop_words = {"the", "is", "a", "an", "and", "it", "of", "to", "in"}

text = "The battery is great. Great battery, great price. The screen is not great."
tokens = [t for t in re.findall(r"[a-z']+", text.lower()) if t not in stop_words]

print(Counter(tokens).most_common(4))

Be careful. “not” is on many stop word lists, and removing it turns “not great” into “great”. What counts as a stop word depends on the task.

Stemming and lemmatisation

“run”, “runs” and “running” are forms of one word. Two techniques reduce words to a common form:

• Stemming cuts off endings by rule. It is fast and crude: “studies” may become “studi”.
• Lemmatisation uses vocabulary and grammar to return the dictionary form: “studies” becomes “study” and “better” becomes “good”.

def simple_stem(word):
    for ending in ("ing", "ed", "es", "s"):
        if word.endswith(ending) and len(word) - len(ending) >= 3:
            return word[: -len(ending)]
    return word

for w in ["running", "jumped", "boxes", "cats", "studies", "is"]:
    print(w, "->", simple_stem(w))

This toy stemmer shows the idea and its weakness. Real projects use a library.

Libraries

On your own computer, NLTK and spaCy provide tested tokenisers, stop word lists, stemmers and lemmatisers.

# pip install spacy
# python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("The studies were running better than expected.")
print([(token.text, token.lemma_, token.pos_) for token in doc])

spaCy also labels each word’s part of speech and finds named entities such as people, places and organisations.

N-grams

A single word loses context. An n-gram is a sequence of n consecutive tokens. Two-word sequences are called bigrams.

tokens = ["the", "screen", "is", "not", "great"]
bigrams = list(zip(tokens, tokens[1:]))
print(bigrams)

The bigram ("not", "great") keeps the meaning that the single words lose.

How much preprocessing?

Classic methods that count words benefit from thorough cleaning. Modern transformer models, covered at the end of this module, have their own tokenisers and expect text that is close to raw. Match the preprocessing to the method.

Practice

Write a function that takes a sentence and returns its three most common words after lowercasing and removing stop words.