P-Values Explained

A p-value is the probability of getting results at least as extreme as yours if the null hypothesis were true. A small p-value means your data would be surprising in a world where nothing is going on. That is all it means, and most misreadings of research come from asking it to mean more.

A coin example

You flip a coin 100 times and get 60 heads. Is the coin biased?

Start by assuming it is fair (H₀: p = 0.5). A fair coin gives 50 heads on average, with a standard deviation of √(100 × 0.5 × 0.5) = 5. Sixty heads is 10 above the average, or 2 standard deviations, so z = 2.0.

The chance that a fair coin lands 2 or more standard deviations away from 50, in either direction, is about 0.046. That is the p-value. A fair coin would give a result this lopsided about 1 time in 22.

Run the coin example in the one-proportion z-test calculator

The calculator also shows the exact binomial p-value, 0.057. The normal approximation and the exact answer fall on opposite sides of 0.05. Borderline results like this are a good reason not to treat 0.05 as a magic line.

What a p-value is not

Wrong readingWhy it’s wrong
“There is a 4.6% chance the coin is fair.”The p-value assumes the coin is fair and asks how surprising the data are. It can’t turn that around and give you the probability the assumption is true.
“There is a 95.4% chance the coin is biased.”Same mistake, other direction.
“p = 0.001 means a bigger effect than p = 0.04.”Not necessarily. A tiny effect in a huge sample can give a tiny p-value. Look at the effect size.
“p = 0.20 proves there is no effect.”It only says the data are not unusual under H₀. A small study can easily miss a real effect.
“p < 0.05 means the result matters.”Statistical significance and practical importance are different things. A diet that cuts 0.2 kg in a year can be significant with enough people.

Where does 0.05 come from?

Mostly from habit. Ronald Fisher suggested it in the 1920s as a convenient point for deciding whether a result was worth a second look, not as a rule for what is true. Some fields use 0.01. Particle physics uses about 0.0000003 (five standard deviations) before claiming a discovery. Pick α before you look at your data, and report the exact p-value rather than just “p < 0.05”.

One-tailed and two-tailed p-values

The coin p-value of 0.046 is two-tailed: it counts lopsided results in both directions, too many heads or too many tails. If you had set out to test only whether the coin favours heads, the one-tailed p-value would be half that, 0.023. Decide which you need before you collect the data.

Getting a p-value from a test statistic

If you already have a z, t, χ² or F value from a textbook or a paper, you can turn it into a p-value directly:

Or bracket it by hand using the t table or chi-square table.

What to report alongside p

A p-value on its own says little about what you found. Report the effect itself (a difference, a correlation, an odds ratio), a confidence interval for it and the sample size. “The new layout raised sign-ups from 4.1% to 5.0% (95% CI for the difference 0.2 to 1.6 points, p = 0.01)” tells the reader far more than “p = 0.01”.

Check yourself

A study reports p = 0.03. A colleague says “so there’s a 97% chance the treatment works”. What would you tell them?

That’s not what p means. It means that if the treatment did nothing, results this strong would appear about 3% of the time. The chance the treatment works depends on other things too, such as how plausible it was before the study.

Two studies test the same drug. One finds p = 0.04, the other p = 0.06. Do they disagree?

Not really. Their results are almost the same, and both sit near the 0.05 line. Compare their effect sizes and confidence intervals, which probably overlap a lot.