Evaluating Models: Precision, Recall and Cross-Validation

Accuracy is the share of predictions that are correct. It is easy to understand and often misleading. If 1% of transactions are fraudulent, a model that never predicts fraud is 99% accurate and completely useless.

The confusion matrix

A confusion matrix counts each kind of outcome.

	Predicted negative	Predicted positive
Actually negative	True negative (TN)	False positive (FP)
Actually positive	False negative (FN)	True positive (TP)

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y   # make malignant the positive class (1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

print(confusion_matrix(y_test, model.predict(X_test)))

Rows are the actual classes and columns are the predicted classes.

Precision, recall and F1

Metric	Formula	Question it answers
Precision	TP / (TP + FP)	Of the cases flagged positive, how many really were?
Recall	TP / (TP + FN)	Of the real positives, how many did we find?
F1	Harmonic mean of the two	One number that balances both

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

print(classification_report(y_test, model.predict(X_test), target_names=["benign", "malignant"]))

Which metric matters depends on the cost of each error:

• When a missed positive is costly, such as a disease or fraud, favour recall.
• When a false alarm is costly, such as blocking a real customer, favour precision.

ROC AUC

The ROC curve plots the true positive rate against the false positive rate at every threshold. The area under the curve (AUC) summarises it in one number: 0.5 is no better than guessing, and 1.0 is perfect. AUC measures how well the model ranks positives above negatives, independent of any one threshold.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
y = 1 - y
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)

print("ROC AUC:", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 4))

When positives are very rare, the precision-recall curve is usually more informative than the ROC curve.

Cross-validation

A single train and test split gives a score that depends on which rows happened to land in the test set. k-fold cross-validation splits the data into k parts, trains k times, each time testing on a different part, and reports every score.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

scores = cross_val_score(model, X, y, cv=5, scoring="f1")
print("F1 in each fold:", scores.round(3))
print(f"mean {scores.mean():.3f}, standard deviation {scores.std():.3f}")

The mean is a more reliable estimate than one split, and the spread shows how stable the model is.

Keep the test set for the end

Use cross-validation on the training data to compare models and settings. Touch the test set once, at the very end, to report the final result. If you choose a model by looking at test scores, the test score is no longer an honest estimate.

Practice

Run the cross-validation with scoring="recall" and then scoring="precision". Which is higher for this model?