Unsupervised Learning: Clustering and PCA

Unsupervised learning works on data with no target column. There is no right answer to predict. The aim is to discover structure.

Clustering with k-means

k-means divides rows into k groups so that each row is close to the centre of its group. It works like this:

1. Pick k starting centres.
2. Assign every row to its nearest centre.
3. Move each centre to the mean of the rows assigned to it.
4. Repeat steps 2 and 3 until the assignments stop changing.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
# visits per month, average spend
customers = np.vstack([
    rng.normal([2, 20], [1, 5], (40, 2)),
    rng.normal([10, 35], [2, 6], (40, 2)),
    rng.normal([5, 120], [1.5, 15], (40, 2)),
])

X = StandardScaler().fit_transform(customers)
kmeans = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)

for label in range(3):
    group = customers[kmeans.labels_ == label]
    print(f"cluster {label}: {len(group)} customers, "
          f"visits={group[:, 0].mean():.1f}, spend={group[:, 1].mean():.0f}")

Always scale before k-means. It uses distances, so a feature measured in large numbers would otherwise dominate.

Choosing k

You must choose the number of clusters. The silhouette score measures how well separated the clusters are, from -1 to 1. Higher is better.

import numpy as np
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
customers = np.vstack([
    rng.normal([2, 20], [1, 5], (40, 2)),
    rng.normal([10, 35], [2, 6], (40, 2)),
    rng.normal([5, 120], [1.5, 15], (40, 2)),
])
X = StandardScaler().fit_transform(customers)

for k in range(2, 7):
    labels = KMeans(n_clusters=k, n_init=10, random_state=0).fit_predict(X)
    print(f"k={k}: silhouette={silhouette_score(X, labels):.3f}")

A score is a guide. The best k is the one whose groups are useful to the people who will act on them.

Limits of k-means

k-means assumes clusters are roughly round and similar in size, and every row is forced into a cluster. For irregular shapes or for detecting noise, look at DBSCAN. For a tree of nested groups, look at hierarchical clustering.

Dimensionality reduction with PCA

Principal component analysis (PCA) replaces many correlated features with a few new ones, called components, that keep as much of the variation as possible. Each component is a weighted combination of the original features.

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)

pca = PCA(n_components=5).fit(X_scaled)
print("share of variance per component:", pca.explained_variance_ratio_.round(3))
print("total kept by 5 components:", round(pca.explained_variance_ratio_.sum(), 3))
print("shape before:", X.shape, " after:", pca.transform(X_scaled).shape)

Thirty features are reduced to five while most of the variation is kept.

What PCA is used for

• Visualisation. Two components can be plotted to see whether groups separate.
• Speed. Fewer features mean faster training.
• Noise reduction. Dropping the smallest components removes minor variation.

The cost is interpretability: a component is a mixture of features, not a measurement you can name.

PCA inside a model

from sklearn.datasets import load_breast_cancer
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
for n in (2, 5, 30):
    model = make_pipeline(StandardScaler(), PCA(n_components=n), LogisticRegression(max_iter=1000))
    print(f"{n:2d} components: accuracy={cross_val_score(model, X, y, cv=5).mean():.3f}")

Practice

Run the k-means example without StandardScaler. Do the clusters still match the three groups the data was generated from?