What is a good p-value for A/B testing?

A p-value less than 0.05 (5%) is generally considered significant, meaning there's less than a 5% chance the difference is due to random chance. This corresponds to 95% confidence level, which is the standard for most A/B tests.

How long should an A/B test run?

Run tests until you reach the minimum sample size calculated upfront. This typically takes 1-4 weeks depending on traffic volume. Don't stop early just because you see significance—this inflates false positive rates.

What's the difference between statistical and practical significance?

Statistical significance tells you if a difference is real (not random). Practical significance tells you if the difference matters for your business. A 0.1% lift might be statistically significant but not worth implementing.

Can I check A/B test results early?

Peeking at results before reaching minimum sample size increases false positive rates. Use sequential testing methods if you need to monitor progress, or wait until the test completes.

What sample size do I need for an A/B test?

Sample size depends on baseline conversion rate, minimum detectable effect (MDE), statistical power (typically 80%), and confidence level (typically 95%). Use online calculators to determine requirements before starting your test.

A/B Testing Statistical Significance: Complete Guide 2025

You've run an A/B test. Variant B shows a 12% lift in conversions. Should you ship it? The answer depends on statistical significance—a concept that separates data-driven decisions from wishful thinking.

Statistical significance tells you whether the difference you're seeing is real or just random noise. Without it, you might roll out changes that actually hurt performance, or miss opportunities that could drive real growth.

This guide explains statistical significance in practical terms: what it means, how to calculate it, when to trust your results, and common pitfalls that trip up even experienced experimenters.

What Is Statistical Significance in A/B Testing?

Statistical significance is a measure of confidence that the difference between your test variants is real, not due to random chance. It answers the question: "If there was no real difference between A and B, how likely is it that we'd see results this different by chance?"

In A/B testing, you're comparing two versions of a page, feature, or experience. Statistical significance helps you determine if the observed difference (e.g., "Variant B converted 12% better") is likely a true effect or just statistical noise.

Data visualization showing A/B test results with statistical significance indicators — Understanding statistical significance helps you make confident decisions from A/B test data

Key Concepts: P-Value and Confidence Level

P-Value

The p-value is the probability of seeing results at least as extreme as yours if there was no real difference between variants. A lower p-value means it's less likely the results are due to chance.

p < 0.05 (5%): Generally considered significant. There's less than a 5% chance the difference is random.
p < 0.01 (1%): Highly significant. Less than 1% chance it's random.
p > 0.05: Not significant. The difference could easily be due to chance.

Confidence Level

Confidence level is the complement of p-value. A 95% confidence level means you're 95% confident the difference is real (p < 0.05). Most A/B testing tools default to 95% confidence, though some experiments use 99% for higher-stakes decisions.

Common Thresholds: 95% confidence (p < 0.05) is standard for most experiments. Use 99% confidence (p < 0.01) for high-impact changes like pricing or core user flows.

How to Calculate Statistical Significance

Most A/B testing platforms calculate significance automatically, but understanding the math helps you interpret results and catch errors.

Basic Formula

Statistical significance testing typically uses a z-test or t-test to compare conversion rates:

z = (p₁ - p₂) / √(p(1-p)(1/n₁ + 1/n₂))

Where:
- p₁ = conversion rate of variant A
- p₂ = conversion rate of variant B  
- p = pooled conversion rate
- n₁ = sample size of variant A
- n₂ = sample size of variant B

The z-score tells you how many standard deviations the difference is from zero. A z-score above 1.96 (for 95% confidence) or 2.58 (for 99% confidence) indicates significance.

Sample Size Requirements

Before running a test, calculate the minimum sample size needed to detect a meaningful difference. Too small a sample, and you won't have enough power to detect real effects. Too large, and you're wasting traffic.

Factors affecting sample size:

Baseline conversion rate: Lower baselines require larger samples
Minimum detectable effect (MDE): The smallest lift you want to detect (e.g., 10% relative increase)
Statistical power: Typically 80% (20% chance of false negatives)
Confidence level: 95% or 99%

Sample size calculator and statistical analysis dashboard for A/B testing — Proper sample size calculation ensures your A/B tests have enough power to detect real differences

Common Statistical Significance Thresholds

Confidence Level	P-Value	Z-Score	When to Use
90%	< 0.10	> 1.65	Low-stakes tests, early exploration
95%	< 0.05	> 1.96	Standard for most experiments
99%	< 0.01	> 2.58	High-impact changes (pricing, core flows)

How to Interpret A/B Test Results

Scenario 1: Significant Winner

Example: Variant B shows 15% lift, p-value = 0.02 (95% confidence)

✅ Action: Ship variant B. The difference is statistically significant.
✅ Confidence: There's only a 2% chance this is random noise.

Scenario 2: Not Significant

Example: Variant B shows 8% lift, p-value = 0.12 (not significant)

⚠️ Action: Don't ship yet. The difference could be random.
⚠️ Options: Run longer to collect more data, or test a different hypothesis.

Scenario 3: Significant Loss

Example: Variant B shows -10% lift, p-value = 0.03 (significant negative)

❌ Action: Don't ship. The variant is significantly worse.
💡 Learning: Document what didn't work and why.

Common Mistakes and How to Avoid Them

1. Peeking at Results Early

Problem: Checking results before reaching minimum sample size inflates false positive rates.

Solution: Set sample size upfront, use sequential testing methods, or wait until the test completes.

2. Stopping When You See Significance

Problem: Stopping as soon as p < 0.05 can lead to false positives due to multiple comparisons.

Solution: Pre-determine test duration or use sequential analysis frameworks.

3. Ignoring Confidence Intervals

Problem: Focusing only on point estimates (e.g., "12% lift") ignores uncertainty.

Solution: Always report confidence intervals (e.g., "12% lift, 95% CI: 8% to 16%").

4. Testing Too Many Variants

Problem: Running A/B/C/D tests without adjusting for multiple comparisons increases false positives.

Solution: Use Bonferroni correction or limit to A/B tests unless using proper multivariate testing frameworks.

5. Ignoring Practical Significance

Problem: A statistically significant 0.1% lift might not be worth implementing.

Solution: Consider both statistical and practical significance. Is the lift meaningful for your business?

A/B testing dashboard showing experiment results with confidence intervals and statistical significance — Proper A/B test analysis includes confidence intervals, not just point estimates

Sample Size Calculators

Use these tools to calculate minimum sample size before running tests:

Optimizely Sample Size Calculator: optimizely.com/sample-size-calculator
Evan Miller's A/B Test Calculator: evanmiller.org/ab-testing
VWO Sample Size Calculator: Built into VWO platform

Real-World Example

Test: New checkout button color (red vs. green)

Baseline (Variant A): 2.5% conversion rate, 10,000 visitors

Test (Variant B): 2.8% conversion rate, 10,000 visitors

Calculation:

Difference: 0.3 percentage points (12% relative lift)
P-value: 0.04 (significant at 95% confidence)
Confidence interval: 8% to 16% relative lift

Decision: Ship variant B. The 12% lift is statistically significant and the confidence interval suggests a meaningful improvement.

Best Practices

Calculate sample size upfront: Know how long your test needs to run before starting.
Use 95% confidence as standard: Reserve 99% for high-stakes decisions.
Report confidence intervals: Don't just say "12% lift"—say "12% lift (95% CI: 8-16%)".
Avoid peeking: Set test duration and stick to it, or use sequential testing methods.
Consider practical significance: A statistically significant 0.5% lift might not be worth the implementation cost.
Document everything: Record hypothesis, sample size, duration, and results for future reference.

Conclusion

Statistical significance is the foundation of trustworthy A/B testing. Without it, you're making decisions based on noise, not signal. By understanding p-values, confidence levels, and sample size requirements, you can run experiments that deliver real, measurable improvements.

Remember: statistical significance tells you if a difference is real. Practical significance tells you if it matters. Use both to make data-driven decisions that drive growth.

If you need help setting up A/B testing frameworks, calculating sample sizes, or interpreting results, get in touch for experimentation consulting.