Decision Trees and Random Forests

Decision trees

A decision tree predicts by asking a sequence of yes or no questions about the features, such as “is the petal shorter than 2.5 cm?”. Each answer leads down a branch until a leaf gives the prediction.

Training chooses, at each step, the question that best separates the classes.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

iris = load_iris()
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(iris.data, iris.target)

print(export_text(tree, feature_names=iris.feature_names))

The printed rules are the whole model. This is the great strength of a small tree: anyone can read it.

Trees need no feature scaling and can capture relationships that are not straight lines.

Trees overfit

Left to grow, a tree keeps splitting until it has memorised the training data.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)

for depth in (1, 3, 5, None):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(X_train, y_train)
    print(f"max_depth={depth}: train={tree.score(X_train, y_train):.3f}  test={tree.score(X_test, y_test):.3f}")

With no depth limit, the training accuracy reaches 1.0 while the test accuracy does not improve. A large gap between the two is the sign of overfitting.

Limit a tree with max_depth, min_samples_leaf or min_samples_split.

Random forests

A random forest trains many trees, each on a random sample of the rows and considering a random subset of the features at each split. The forest predicts by majority vote, or by averaging for regression. The errors of individual trees tend to cancel out.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)

for name, model in [
    ("single tree", DecisionTreeClassifier(random_state=0)),
    ("random forest", RandomForestClassifier(n_estimators=100, random_state=0)),
]:
    scores = cross_val_score(model, X, y, cv=5)
    print(f"{name:14s} accuracy: {scores.mean():.3f}")

Feature importance

A forest reports how much each feature contributed to its splits.

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

data = load_breast_cancer()
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(data.data, data.target)

importance = pd.Series(forest.feature_importances_, index=data.feature_names)
print(importance.sort_values(ascending=False).head(6).round(3))

Importances show what the model used, not what causes the outcome. When two features are correlated, the importance is shared between them.

Gradient boosting

Gradient boosting also combines many trees, but builds them one after another, each correcting the errors of those before it. Boosted trees are often the most accurate models for tabular data. scikit-learn provides HistGradientBoostingClassifier, and the XGBoost and LightGBM libraries are widely used.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score

X, y = load_breast_cancer(return_X_y=True)
scores = cross_val_score(HistGradientBoostingClassifier(random_state=0), X, y, cv=5)
print("gradient boosting accuracy:", round(scores.mean(), 3))

Choosing between them

Model	Strength	Weakness
Single tree	Easy to explain	Overfits, unstable
Random forest	Accurate with little tuning	Harder to explain, larger
Gradient boosting	Often the most accurate	More settings to tune

Practice

Train a random forest with n_estimators set to 5, 50 and 300. At what point does adding trees stop improving the cross-validated accuracy?