The right test depends on three things: what kind of outcome you measured, how many groups or variables you are comparing, and whether the same people appear more than once. Answer the questions below to get a recommendation, or use the tables further down.
Answer a few questions and we’ll point you to the right test.
Step 1: what kind of outcome do you have?
- A measurement on a scale, such as weight, time, score or income. Means and standard deviations make sense.
- A category, such as yes/no, pass/fail or blood type. You count how many fall into each group.
- A rank or rating on a short scale, such as a 1 to 5 satisfaction score. These are often treated as measurements if there are enough points on the scale, but rank-based tests are safer when the scale is short or the data are lopsided.
Step 2: find your situation in the table
Comparing groups on a measurement
| Situation | Usual test | If data are skewed or have outliers |
|---|---|---|
| One group against a known value | One-sample t-test | Wilcoxon signed-rank |
| Two separate groups | Independent t-test (Welch) | Mann-Whitney U |
| Same people measured twice | Paired t-test | Wilcoxon signed-rank |
| Three or more separate groups | One-way ANOVA | Kruskal-Wallis |
| Two grouping factors | Two-way ANOVA | Consider transforming the data |
Counts and categories
| Situation | Test |
|---|---|
| One proportion against a known value | One-proportion z-test |
| Two proportions, such as an A/B test | Two-proportion z-test or A/B test calculator |
| Two categorical variables in a table | Chi-square test of independence |
| A 2 × 2 table with small counts | Fisher’s exact test |
| Counts in categories against expected shares | Chi-square goodness of fit |
Relationships between variables
| Situation | Test |
|---|---|
| Two measurements, straight-line relationship | Pearson correlation |
| Two measurements or ranks, any steady upward or downward trend | Spearman correlation |
| Predict a measurement from one variable | Simple linear regression |
| Predict a measurement from several variables | Multiple regression |
| Predict a yes/no outcome | Logistic regression |
Step 3: paired or independent?
This is the question people get wrong most often. If the same people or items appear in more than one group, or each value in one group is matched to one value in the other, the data are paired. Using an independent test on paired data throws away information, and the result can flip from significant to not significant. The paired t-test guide has an example where exactly that happens.
A note on normality
“Use a non-parametric test if the data aren’t normal” is often over-applied. The t-test and ANOVA hold up well with moderate skew once you have around 30 per group. Worry more about clear outliers and very small samples. With 8 values per group and one wild value, a rank-based test is the safer choice.
Check yourself
You record whether 200 website visitors clicked a button, split between two page designs. Which test?
The outcome is yes/no and there are two independent groups, so a two-proportion z-test, or the chi-square test on the 2 × 2 table. They give the same p-value.
Twelve patients rate their pain from 0 to 10 before and after a treatment. Which test?
Same patients twice, so a paired test. With a short rating scale and only 12 people, the Wilcoxon signed-rank test is a sensible choice. A paired t-test is also commonly used.