Independent Samples T-Test: Formulas, Assumptions and Example

The independent-samples t-test compares the means of two separate groups, where nobody is in both. Examples: a treatment group and a control group, men and women, two different schools.

A worked example

Ten students learn a topic with method A and ten different students use method B. Their test scores:

Method AMethod B
78, 85, 72, 90, 81, 76, 88, 83, 79, 8470, 75, 68, 80, 72, 74, 69, 77, 71, 73
Mean 81.6, SD 5.52Mean 72.9, SD 3.73

The difference in means is 8.7 points. The question is whether a gap that size could come from chance alone, given how much scores vary within each group.

  1. Standard error of the difference: √(5.52²/10 + 3.73²/10) = 2.11.
  2. t = 8.7 ÷ 2.11 = 4.13.
  3. Welch’s degrees of freedom come out at 15.8, which gives p = 0.0008.
  4. The 95% confidence interval for the difference is 4.2 to 13.2 points.

Run the teaching-method example in the calculator

Method A students scored higher, and the gap is too large to put down to chance. Cohen’s d is 1.85, a very large effect.

Welch’s or Student’s t-test?

There are two versions. Student’s t-test assumes both groups have the same variance and pools them. Welch’s t-test doesn’t make that assumption.

Use Welch’s unless you have a good reason not to. When the variances really are equal, Welch’s gives almost the same answer as Student’s. When they are not, and especially when the group sizes differ, Student’s can give badly wrong p-values. Many statisticians now recommend Welch’s as the default, and it is the default in R.

For the example above, Student’s version gives t = 4.13, df = 18, p = 0.0006. Practically the same answer.

Assumptions

  • Independence. Each person is in one group only, and one person’s score doesn’t affect another’s. Students who studied together are a problem.
  • Roughly normal data in each group. With 30 or more per group this matters little. With small groups and obvious skew or outliers, use the Mann-Whitney U test.

Reporting

“Students taught with method A (M = 81.6, SD = 5.5) scored higher than those taught with method B (M = 72.9, SD = 3.7), Welch’s t(15.8) = 4.13, p < .001, d = 1.85, 95% CI for the difference [4.2, 13.2].”

Common mistakes

  • Using this test on before-and-after data. If the same people are in both columns, use the paired t-test.
  • Running a separate t-test for every pair of three or more groups. With three groups that is three tests, and the chance of at least one false positive climbs to about 14%. Use one-way ANOVA.
  • Reading “not significant” as “no difference”. With small groups, even a real difference can fail to reach significance. Look at the confidence interval to see what differences your data can’t rule out.

Check yourself

Group 1: n = 25, mean 50, SD 10. Group 2: n = 25, mean 44, SD 10. Roughly what is t?

SE = √(100/25 + 100/25) = √8 = 2.83. t = 6 ÷ 2.83 = 2.12, with df = 48. That gives p ≈ 0.039, just under 0.05.