Overfitting, Underfitting and Hyperparameter Tuning

A model is useful only if it works on data it has not seen. Two opposite problems get in the way.

	Underfitting	Overfitting
The model is	Too simple	Too complex
Training score	Poor	Excellent
Test score	Poor	Much worse than training
It has	Missed the pattern	Memorised noise

Seeing both

This example fits polynomial curves of increasing degree to noisy data.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
X = rng.uniform(-3, 3, 60).reshape(-1, 1)
y = np.sin(X).ravel() + rng.normal(0, 0.3, 60)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

for degree in (1, 4, 15):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X_train, y_train)
    print(f"degree {degree:2d}: train R2={model.score(X_train, y_train):.3f}  test R2={model.score(X_test, y_test):.3f}")

Degree 1 is a straight line and underfits a curve. Degree 15 follows the training points closely and does worse on the test set. The middle setting generalises best.

The bias-variance trade-off

• Bias is error from a model that is too simple to capture the pattern.
• Variance is error from a model that changes a lot when the training data changes.

Making a model more flexible lowers bias and raises variance. The goal is the point where the total error on new data is lowest.

Ways to reduce overfitting

• Get more training data.
• Use a simpler model, or limit its complexity, such as tree depth.
• Add regularisation, which penalises complexity.
• Remove features that carry no signal.
• Combine many models, as a random forest does.

Hyperparameters

A parameter is learned from the data, such as a coefficient. A hyperparameter is a setting you choose before training, such as the depth of a tree or the strength of a penalty.

Grid search

GridSearchCV tries every combination of the values you list, scores each with cross-validation and keeps the best.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)

grid = GridSearchCV(
    RandomForestClassifier(n_estimators=50, random_state=0),
    param_grid={"max_depth": [3, 6, None], "min_samples_leaf": [1, 5]},
    cv=3,
    scoring="f1",
)
grid.fit(X_train, y_train)

print("best settings:", grid.best_params_)
print("best cross-validated F1:", round(grid.best_score_, 3))
print("F1 on the test set:", round(grid.score(X_test, y_test), 3))

The search uses only the training data. The test set is scored once at the end.

To tune a step inside a pipeline, name the setting as step__setting, for example classifier__C.

Random search

A grid grows quickly: four settings with five values each is 625 combinations. RandomizedSearchCV samples a fixed number of combinations and usually finds a good one with much less computation.

A sensible order of work

1. Get a simple baseline working end to end.
2. Improve the data and features.
3. Try two or three model families with default settings.
4. Tune the best one.

Tuning rarely rescues a model built on poor features.

Practice

In the polynomial example, try degrees 2, 3, 6 and 10. Which degree gives the best test score?