Classification with Logistic Regression

Classification predicts which category a row belongs to: spam or not, will churn or will stay. Logistic regression is the standard first model. Despite its name, it is a classification method.

How it works

Logistic regression calculates a weighted sum of the features, as linear regression does, and passes it through the sigmoid function, which squeezes any number into the range 0 to 1. The result is read as the probability of the positive class.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in (-4, -1, 0, 1, 4):
    print(z, "->", round(sigmoid(z), 3))

Train a classifier

The breast cancer dataset has 30 measurements of cell samples. The target is 0 for malignant and 1 for benign.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)

print("accuracy on the test set:", round(model.score(X_test, y_test), 3))

Why scale the features

The features in this dataset have very different ranges. StandardScaler subtracts the mean and divides by the standard deviation, so that every feature has mean 0 and standard deviation 1. Models that rely on distances or on penalised coefficients train faster and more reliably on scaled data.

make_pipeline chains the scaler and the model. The scaler learns its means and standard deviations from the training data only, and then applies them to the test data.

Probabilities

predict returns a class. predict_proba returns the probability of each class.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

probabilities = model.predict_proba(X_test[:5])
print(probabilities.round(3))
print("predicted classes:", model.predict(X_test[:5]))

The decision threshold

By default a row is assigned to class 1 when its probability is at least 0.5. You can choose a different threshold. In a medical setting, missing a malignant case is far worse than a false alarm, so you might flag a sample as malignant unless the model is very sure that it is benign.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

p_benign = model.predict_proba(X_test)[:, 1]
for threshold in (0.5, 0.9):
    predicted_benign = p_benign >= threshold
    missed = ((y_test == 0) & predicted_benign).sum()
    false_alarms = ((y_test == 1) & ~predicted_benign).sum()
    print(f"threshold {threshold}: malignant cases missed={missed}, false alarms={false_alarms}")

Raising the threshold trades one kind of error for the other. The next lesson gives you the vocabulary to measure that trade.

More than two classes

Logistic regression handles several classes too. scikit-learn does this automatically when the target has more than two values.

Practice

Remove StandardScaler() from the pipeline and train again. Does the accuracy change? Do you see a warning about convergence if you lower max_iter to 100?