What is statistical significance in A/B testing?

Statistical significance tells you how confident you can be that your test results aren't due to random chance. A 95% significance level means there's only a 5% chance the difference you're seeing is random noise. But here's what most people miss. 95% confidence doesn't mean 95% chance your variant is better. It means if there was NO real difference, you'd see results this extreme only 5% of the time.

Why do most A/B tests give wrong conclusions?

Three main reasons. 1) Peeking, checking results daily and stopping when you see a 'winner' inflates false positives from 5% to 30%+. 2) Underpowered tests, not enough sample size means you miss real effects or see fake ones. 3) Multiple comparisons, testing many variants without correction means some will 'win' by chance. Our calculator warns you about all three.

How many visitors do I need for an A/B test?

It depends on your baseline conversion rate and minimum detectable effect (MDE). For example: 3% baseline CVR, 10% relative lift (to 3.3%), 80% power, 95% significance = ~25,000 visitors per variant. Want to detect a 5% lift? You need ~100,000 per variant. Our calculator shows exactly how long your test needs to run based on your traffic.

What's the difference between confidence and power?

Confidence (typically 95%) is about avoiding false positives, calling a winner when there isn't one. Power (typically 80%) is about avoiding false negatives, missing a real winner. Low power means you might run a test, see 'no significant difference,' and abandon a variant that was actually 15% better. Both matter, but most tools only show confidence.

Should I use Bayesian or Frequentist A/B testing?

Frequentist is stricter and better for high-stakes decisions where false positives are costly. Bayesian is more intuitive (gives you 'probability variant B is better') and allows continuous monitoring. For most marketing tests, either works if used correctly. The bigger issue is usually sample size and peeking, not methodology.

Can I check my A/B test results daily?

With traditional frequentist testing, no. Each peek inflates your false positive rate. If you peek daily on a 2-week test, your 'real' significance is closer to 70-80%, not 95%. Solutions: 1) Set a fixed end date and don't look until then, 2) Use Bayesian methods which handle continuous monitoring better. Our calculator detects early stopping and applies a peeking penalty to warn you.

My test shows 94% significance, should I ship it?

Probably not yet. 94% means 6% chance you're shipping a worse experience, that's 1 in 17. More importantly, ask: What's the cost of being wrong vs. the cost of waiting? If shipping a loser costs you $50K/year and waiting 3 more days gets you to 97% confidence, wait. Our calculator shows the expected value math to make this decision clear.

Now in pilot: Agent Commerce Protocol, the open standard for AI commerce.Pilot live: ACP for AI commerce.Explore ACP

Test significance and revenue impact.

Calculate significance from your control and variant data. See power, lift, and modelled revenue impact at your traffic.

SamStrategy

Try a sample

Test name (optional)

Control (A)

Visitors *Conversions *

Variant B (challenger)

Visitors *Conversions *

Statistical significance via two-proportion z-test, statistical power via the standard normal approximation, MDE via inversion of the power equation, and revenue impact as point-estimate lift multiplied by your monthly visitors and AOV.