Start Learning Contact Us
Data Science with Python

Feature Engineering and Pipelines

Machine Learning · Lesson 18 of 30

Models need a table of numbers with no gaps. Real data has missing values, text categories and columns on wildly different scales. Preparing it well usually improves a model more than switching algorithms.

A realistic dataset

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "age": [34, 45, np.nan, 23, 51, 38, 29, np.nan, 47, 31],
    "income": [42, 81, 55, np.nan, 97, 60, 39, 48, 88, 52],
    "plan": ["basic", "pro", "basic", "basic", "pro", "team", "basic", "team", "pro", "basic"],
    "churned": [1, 0, 1, 1, 0, 0, 1, 0, 0, 1],
})
print(df)
print(df.isna().sum())

Missing values

SimpleImputer fills gaps with a statistic from the column.

import numpy as np
from sklearn.impute import SimpleImputer

ages = np.array([[34], [45], [np.nan], [23], [51]])
print(SimpleImputer(strategy="median").fit_transform(ages).ravel())

Use the median for numbers and the most frequent value for categories. The median is less affected by outliers than the mean.

Encoding categories

A model cannot use the text “basic” or “pro”. One-hot encoding creates one 0-or-1 column for each category.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

plans = pd.DataFrame({"plan": ["basic", "pro", "team", "basic"]})
encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
print(encoder.fit_transform(plans))
print(encoder.get_feature_names_out())

handle_unknown="ignore" stops the model failing when it meets a category it did not see in training.

Do not replace categories with arbitrary numbers such as basic=1, pro=2, team=3 for a linear model. That tells the model that team is three times basic.

Scaling

StandardScaler gives each numeric feature mean 0 and standard deviation 1. Linear models, k-nearest neighbours and neural networks need this. Tree-based models do not.

Putting it together

ColumnTransformer applies different steps to different columns, and Pipeline chains preparation and model into one object.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

df = pd.DataFrame({
    "age": [34, 45, np.nan, 23, 51, 38, 29, np.nan, 47, 31] * 6,
    "income": [42, 81, 55, np.nan, 97, 60, 39, 48, 88, 52] * 6,
    "plan": ["basic", "pro", "basic", "basic", "pro", "team", "basic", "team", "pro", "basic"] * 6,
    "churned": [1, 0, 1, 1, 0, 0, 1, 0, 0, 1] * 6,
})
X, y = df.drop(columns="churned"), df["churned"]

numeric = Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())])
categorical = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                        ("encode", OneHotEncoder(handle_unknown="ignore"))])

prepare = ColumnTransformer([
    ("numeric", numeric, ["age", "income"]),
    ("categorical", categorical, ["plan"]),
])
model = Pipeline([("prepare", prepare), ("classifier", LogisticRegression())])

print("cross-validated accuracy:", cross_val_score(model, X, y, cv=5).mean().round(3))

This tiny dataset repeats the same ten customers, so the score is not meaningful. The point is the structure, which works unchanged on real data.

Data leakage

Leakage happens when information that would not be available at prediction time reaches the model during training. The model looks excellent in testing and fails in use.

Two common causes:

  • Preparing before splitting. If you scale or impute using the whole dataset, statistics from the test rows leak into training. A pipeline prevents this, because each cross-validation fold fits the preparation steps on its own training part only.
  • Features that reveal the answer. A “cancellation date” column perfectly predicts churn, but it only exists after the customer has churned.

If a model’s score seems too good to be true, look for leakage first.

Creating features

New columns that express what you know about the problem often help most:

  • Ratios, such as spend per visit.
  • Parts of a date, such as the day of the week or days since signup.
  • Counts and averages over a customer’s history.
  • Groups of a numeric value, such as age bands.

Practice

Add a column income_per_year_of_age to the DataFrame and include it in the numeric columns of the pipeline.