Start Learning Contact Us
Data Science with Python

Correlation and Simple Linear Regression

Statistics · Lesson 12 of 30

Correlation

The Pearson correlation coefficient, written r, measures the strength and direction of a straight-line relationship between two numeric variables. It ranges from -1 to 1.

r Meaning
near 1 As one rises, the other rises
near -1 As one rises, the other falls
near 0 No straight-line relationship 
import numpy as np
from scipy import stats

rng = np.random.default_rng(2)
ad_spend = rng.uniform(1, 10, 60)                     # thousands
sales = 20 + 6 * ad_spend + rng.normal(0, 6, 60)      # thousands

r, p_value = stats.pearsonr(ad_spend, sales)
print("r:", round(r, 3), " p-value:", p_value)

A correlation matrix

pandas calculates r for every pair of columns.

import numpy as np
import pandas as pd

rng = np.random.default_rng(2)
df = pd.DataFrame({"ad_spend": rng.uniform(1, 10, 60)})
df["sales"] = 20 + 6 * df["ad_spend"] + rng.normal(0, 6, 60)
df["temperature"] = rng.normal(22, 5, 60)

print(df.corr().round(2))

Correlation is not causation

Two variables can move together because one causes the other, because a third factor drives both, or by coincidence. Ice cream sales and sunburn rise together because of hot weather, not because of each other. Only a controlled experiment, or careful causal analysis, can show cause.

Correlation also measures only straight-line relationships. A perfect curve can have r near zero:

import numpy as np
from scipy import stats

x = np.linspace(-5, 5, 101)
y = x ** 2
print("r:", round(stats.pearsonr(x, y)[0], 3))

Plot your data before trusting a correlation.

Simple linear regression

Regression goes a step further and fits a line:

y = intercept + slope * x

The line is chosen to minimise the sum of the squared vertical distances between the points and the line. This method is called least squares.

import numpy as np
from scipy import stats

rng = np.random.default_rng(2)
ad_spend = rng.uniform(1, 10, 60)
sales = 20 + 6 * ad_spend + rng.normal(0, 6, 60)

result = stats.linregress(ad_spend, sales)
print("slope:    ", round(result.slope, 2))
print("intercept:", round(result.intercept, 2))
print("R squared:", round(result.rvalue ** 2, 3))
print("p-value for the slope:", result.pvalue)

print("predicted sales at spend 7:", round(result.intercept + result.slope * 7, 1))

Reading the output

  • The slope is the change in y for a one-unit increase in x. Here, each extra thousand spent on advertising is associated with about six thousand more in sales.
  • The intercept is the predicted y when x is zero.
  • R squared is the share of the variation in y that the line explains, from 0 to 1.
  • The p-value tests whether the slope differs from zero.

Residuals

A residual is an actual value minus the predicted value. If the model fits well, the residuals look like random scatter around zero. A pattern in the residuals means a straight line is the wrong shape.

import numpy as np
from scipy import stats

rng = np.random.default_rng(2)
x = rng.uniform(1, 10, 60)
y = 20 + 6 * x + rng.normal(0, 6, 60)

fit = stats.linregress(x, y)
residuals = y - (fit.intercept + fit.slope * x)
print("mean of residuals:", round(residuals.mean(), 6))
print("standard deviation of residuals:", round(residuals.std(ddof=2), 2))

Do not extrapolate

A line fitted to spending between 1 and 10 says nothing reliable about spending of 100. Predictions are only trustworthy inside the range of the data.

This lesson fitted a line with one input. The machine learning module that follows extends the same idea to many inputs and to predicting categories.

Practice

Change the noise in the example from rng.normal(0, 6, 60) to rng.normal(0, 25, 60). What happens to r and to R squared, and why?