A/B Test Statistical Significance Calculator

Quickly calculate the statistical significance and observed power of your A/B tests.

Group A(Control)

Conversion Rate
1.00%

Group B(Test)

Conversion Rate
1.10%
Test Type
One-sided: Detects if Group B is better than Group A. Cannot detect negative impact.
Two-sided: Detects if Group B is better OR worse than Group A. This is the safest, most rigorous standard for reliable reporting.
The probability that a detected lift is not due to random chance. 95% is the industry standard (a 5% risk of a false positive), while 90% is often used for lower-risk changes.

How to use the statistical significance calculator

Already have results? This calculator tells you whether the difference between your two variations is statistically significant, and whether the test was powered enough to trust. It runs entirely in your browser.

  1. 1

    Enter your control group. Add the audience size and number of conversions for Group A (your original).

  2. 2

    Enter your test group. Add the audience size and conversions for Group B (the variation).

  3. 3

    Pick the test direction and confidence. Choose a one-sided or two-sided test and your target confidence level (commonly 90% or 95%).

  4. 4

    Read the verdict. You get the observed lift, z-score, p-value, statistical confidence, and observed power, plus a plain-language read on whether the result is significant and adequately powered.

Key terms explained

P-value
The probability of seeing a difference at least this large if there were truly no effect. A p-value below your threshold (e.g. 0.05) means the result is statistically significant.
Statistical confidence
1 minus the p-value, expressed as a percentage. 95% confidence corresponds to a p-value of 0.05.
Z-score
How many standard errors apart the two conversion rates are. Larger magnitudes mean a more decisive difference.
Observed power
An after-the-fact estimate of how likely the test was to detect the effect it saw. Low observed power flags an underpowered, hard-to-trust result.
Observed lift
The relative change in conversion rate from the control to the variation.
One-sided vs two-sided
A one-sided test only checks whether the variation is better; a two-sided test checks for a difference in either direction and is the safer default.

How significance is calculated

The calculator runs a two-proportion z-test using the pooled standard error. It computes the z-score from the difference in conversion rates, converts it to a p-value for your chosen test direction, and reports the confidence and observed power.

To avoid the most common mistakes, including stopping a test the moment it looks significant, read the guide on the peeking problem, test direction, and inconclusive results.

Frequently asked questions

When is an A/B test result statistically significant?

When the p-value falls below your chosen threshold. A p-value under 0.05 corresponds to 95% confidence that the observed difference is not due to chance.

What does the p-value actually mean?

It is the probability of observing a difference at least as large as yours if the two variations truly performed the same. Smaller is stronger evidence of a real effect.

My result is significant but observed power is low. Can I trust it?

Be cautious. Low observed power suggests the test was underpowered, which raises the chance the significant result is a false positive. Confirm with a properly sized follow-up.

Should I use a one-sided or two-sided test?

Two-sided is the safe default. Use one-sided only when you genuinely care about movement in a single direction and would treat a negative result the same as no result.

Can I stop the test as soon as it shows significance?

No. Repeatedly checking and stopping early (peeking) inflates your false-positive rate well beyond 5%. Decide your sample size in advance and wait for it.

Related guides and tools