Start Learning Contact Us
Data Science with Python

Hypothesis Testing

Statistics · Lesson 11 of 30

A new checkout page has a higher average order value than the old one. Is the new page better, or did it get lucky? Hypothesis testing gives a disciplined way to answer.

The logic

  1. State the null hypothesis: there is no real difference.
  2. State the alternative hypothesis: there is a difference.
  3. Choose a significance level, written alpha, before looking at the data. 0.05 is a common choice.
  4. Calculate a p-value: the probability of seeing a difference at least as large as yours if the null hypothesis were true.
  5. If the p-value is below alpha, reject the null hypothesis.

A p-value is not the probability that the null hypothesis is true. It describes how surprising the data would be if there were no effect.

Comparing two means: the t-test

import numpy as np
from scipy import stats

rng = np.random.default_rng(5)
old_page = rng.normal(loc=50, scale=15, size=200)
new_page = rng.normal(loc=54, scale=15, size=200)

t_stat, p_value = stats.ttest_ind(new_page, old_page, equal_var=False)

print("old mean:", round(old_page.mean(), 2))
print("new mean:", round(new_page.mean(), 2))
print("t statistic:", round(t_stat, 2))
print("p-value:", round(p_value, 4))
print("significant at 0.05:", p_value < 0.05)

equal_var=False runs Welch’s t-test, which does not assume that the two groups have the same variance. It is the safer default.

Use stats.ttest_rel when the two sets of measurements come from the same subjects, such as before and after.

Comparing proportions: the chi-square test

For counts in categories, such as how many visitors bought on each page, use a chi-square test on the table of counts.

from scipy import stats

#            bought  did not buy
table = [[   120,     880],    # old page
         [   150,     850]]    # new page

chi2, p_value, dof, expected = stats.chi2_contingency(table)
print("conversion old:", 120 / 1000, " new:", 150 / 1000)
print("p-value:", round(p_value, 4))

The new page converted 15% of visitors against 12%, yet the p-value is just above 0.05. With 1,000 visitors per page, a difference of this size could still plausibly be chance, so you would keep the test running before deciding.

Two kinds of error

The null hypothesis is true The null hypothesis is false
You reject it Type I error (false positive) Correct
You do not reject it Correct Type II error (false negative)

Alpha is the rate of Type I errors you accept. Power is the probability of detecting a real effect, and it rises with sample size and with the size of the effect.

Significant is not the same as important

With enough data, a tiny difference becomes statistically significant. Always report the size of the effect and a confidence interval, not only the p-value.

import numpy as np
from scipy import stats

rng = np.random.default_rng(8)
a = rng.normal(50.0, 15, 200_000)
b = rng.normal(50.3, 15, 200_000)

print("difference in means:", round(b.mean() - a.mean(), 2))
print("p-value:", stats.ttest_ind(b, a, equal_var=False).pvalue)

The difference is tiny, yet the p-value is far below 0.05.

Common mistakes

  • Testing many things and reporting the one that worked. With 20 tests at alpha 0.05, one false positive is expected by chance alone.
  • Stopping the test as soon as the result looks significant. Decide the sample size in advance.
  • Treating “not significant” as “no effect”. You may simply not have enough data.

Practice

Change the new page’s mean from 54 to 51 and run the t-test again. Is the result still significant with 200 visitors per page? What about with 2,000?