A/B Test Significance Calculator

Find out whether the difference between two variants of a page, email or ad is real or just noise. Enter visitors and conversions for the control (A) and the variant (B) to get both conversion rates, the relative uplift, the p-value, a confidence interval for the difference and, if the test isn’t significant yet, the sample size you need.

How the A/B test works

The calculator runs a two-proportion z-test. It pools both groups to estimate the conversion rate under the assumption that A and B perform the same. Then it measures how many standard errors apart the two observed rates are. The confidence interval for the difference uses Newcombe’s method, which stays accurate when conversion rates are low.

z = (p_B − p_A) ÷ √[ p(1 − p)(1/n_A + 1/n_B) ], p = (x_A + x_B) ÷ (n_A + n_B)

Reading the result

  • Relative uplift: how much better (or worse) B converts than A, as a percentage of A’s rate. Going from 8.0% to 9.3% is a 16.25% uplift.
  • p-value: below 0.05 means a difference this large would be unlikely if A and B truly performed the same.
  • Confidence interval: the plausible range of the true difference in percentage points. If it includes 0, the test hasn’t shown a winner.

Common A/B testing mistakes

  1. Stopping when it first looks significant. Checking results every day and stopping at the first p < 0.05 inflates false positives dramatically. Fix the sample size first with the power analysis calculator (choose “Two proportions”).
  2. Testing too many variants at once without correcting for multiple comparisons.
  3. Running for less than a full business cycle. Weekday and weekend visitors behave differently, so run whole weeks.
  4. Ignoring practical significance. A significant 0.1% uplift may not be worth the cost of the change.

Frequently asked questions

One-sided or two-sided?

Two-sided is the safer default because it also detects a variant that performs worse. Use one-sided only if you decided in advance that you would act only on an improvement.

Is this the same as a chi-square test?

Yes. For two groups, the two-sided z-test is mathematically identical to a 2×2 chi-square test without continuity correction (z² = χ²).