Type I vs Type II Errors and Statistical Power

Every hypothesis test can go wrong in two ways. A Type I error is a false alarm: you conclude there is an effect when there isn’t. A Type II error is a miss: there is a real effect and your test fails to find it.

H₀ is actually trueH₀ is actually false
You reject H₀Type I error (false positive). Probability α.Correct. Probability 1 − β, called power.
You don’t reject H₀Correct. Probability 1 − α.Type II error (false negative). Probability β.

Examples that make it stick

  • Fire alarm. Type I: the alarm goes off when you burn toast. Type II: the alarm stays silent during a real fire.
  • Medical test. Type I: a healthy person is told they have the disease. Type II: a sick person is told they are fine.
  • Court case, where H₀ is “innocent”. Type I: an innocent person is convicted. Type II: a guilty person walks free.

Which error is worse depends on the situation. A missed cancer is usually worse than a false alarm that leads to a second test. Convicting an innocent person is treated as worse than letting a guilty one go, which is why courts demand proof beyond reasonable doubt, in effect a very small α.

You control α directly

When you choose α = 0.05, you are accepting that if H₀ is true, you will wrongly reject it 5% of the time. Lower α to 0.01 and false alarms drop to 1%.

The catch is that lowering α makes Type II errors more likely. A stricter bar means some real effects fail to clear it. The only way to cut both errors at once is to collect more data.

β depends on your sample size and the effect

You can’t set β the way you set α, because it depends on how big the real effect is. But you can plan for it. Say you are comparing two groups and the true difference is half a standard deviation (Cohen’s d = 0.5), using α = 0.05:

People per groupPower (1 − β)Chance of missing the effect (β)
200.3466%
300.4852%
640.8020%

With 30 per group, a study of a real, medium-sized effect is roughly a coin toss. That is why so many small studies “find nothing”. The usual target is 80% power. The power analysis calculator will tell you the power for any design, or the sample size you need.

Running many tests raises the Type I rate

α = 0.05 applies to each test. Run 20 independent tests on data where nothing is going on, and the chance of at least one false positive is 1 − 0.95²⁰ = 64%. This is why ANOVA is used instead of many t-tests, and why post-hoc tests and corrections such as Bonferroni exist.

Check yourself

A drug has no real effect, but the trial reports p = 0.03 and the drug is approved. Which error is that?

Type I. H₀ (no effect) was true and it was rejected.

A study with 15 people per group finds p = 0.21 and concludes the treatment doesn’t work. What should you worry about?

A Type II error. With 15 per group, power is low unless the effect is very large, so a real effect could easily have been missed. Look at the confidence interval for the size of effect the data can’t rule out.

If you lower α from 0.05 to 0.01 and keep the same sample size, what happens to β?

β goes up. You make fewer false alarms but miss more real effects.