Linear Regression

Linear regression predicts a number as a weighted sum of the features:

prediction = intercept + w1 * x1 + w2 * x2 + ...

Training finds the weights, called coefficients, that minimise the squared error on the training data.

Train a model

The diabetes dataset has ten measurements for 442 patients. The target is a measure of disease progression one year later.

import pandas as pd   # needed for as_frame=True
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

model = LinearRegression().fit(X_train, y_train)
print("R squared on the test set:", round(model.score(X_test, y_test), 3))

Measuring error

import pandas as pd   # needed for as_frame=True
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
predictions = LinearRegression().fit(X_train, y_train).predict(X_test)

print("MAE: ", round(mean_absolute_error(y_test, predictions), 1))
print("RMSE:", round(np.sqrt(mean_squared_error(y_test, predictions)), 1))
print("R2:  ", round(r2_score(y_test, predictions), 3))
print("target ranges from", y.min(), "to", y.max())

Metric	Meaning
MAE (mean absolute error)	The average size of the error, in the units of the target
RMSE (root mean squared error)	Like MAE, but large errors count for more
R squared	The share of the variation in the target that the model explains

Always read an error next to the range of the target. An error of 40 means something very different on a scale of 0 to 50 than on a scale of 0 to 5,000.

Reading the coefficients

import pandas as pd
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression

X, y = load_diabetes(return_X_y=True, as_frame=True)
model = LinearRegression().fit(X, y)

coefficients = pd.Series(model.coef_, index=X.columns).sort_values()
print(coefficients.round(1))
print("intercept:", round(model.intercept_, 1))

A coefficient is the change in the prediction for a one-unit increase in that feature, with the other features held fixed. Coefficients can only be compared with each other when the features are on the same scale, which is the case in this dataset.

A coefficient describes an association in the data. It does not show that changing the feature would change the outcome.

Assumptions and limits

Linear regression works best when:

the relationship between each feature and the target is roughly a straight line,
the errors have similar spread across the range of predictions,
the features are not strongly correlated with each other. When they are, individual coefficients become unstable, although predictions can still be good.

Regularisation

With many features, a linear model can fit noise. Ridge and Lasso regression add a penalty on large coefficients, which usually improves results on new data. Lasso can shrink some coefficients to exactly zero, which removes those features from the model.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import cross_val_score

X, y = load_diabetes(return_X_y=True)
for name, model in [("linear", LinearRegression()), ("ridge", Ridge(alpha=0.1)), ("lasso", Lasso(alpha=0.1))]:
    score = cross_val_score(model, X, y, cv=5, scoring="r2").mean()
    print(f"{name:7s} mean R2: {score:.3f}")

alpha sets the strength of the penalty.

Practice

Train the model using only the bmi column as the feature. How much lower is the test R squared than with all ten features?